Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

341–350 of 604 posts

Re: We were wrong about GPUs

#341

Post by figassis »

remram wrote: post #200
BobbyTables2 wrote: post #179

Earlier quoted context omitted.

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

I use k8s, wouldn't call it simple, but there are ways to minimize the complexity of your setup. Mostly, what devs see as complexity is k8s packages a lot of system fundamentals, like networking, storage, name resolution, distributed architectures, etc, and if you mainly spent your career in a single lane, k8s becomes impossible to grasp. Not saying those devs are wrong, not everyone needs to be a networking pro.

K8s is meant to be operated by some class of engineers, and used by another. Just like you have DBAs, sysadmins, etc, maybe your devops should have more system experience besides terraform.

Re: We were wrong about GPUs

#342

Post by whilenot-dev »

raziel2p wrote: post #306
Rury wrote: post #219

Earlier quoted context omitted.

Yeah, I remember watching a video of (I think?) a European professor who helped with an issue devs were having in developing The Witness. Turns out they had a large algorithm they developed in high level code (~2000 lines of code? can't remember) to place flora in the game world, which took minutes to process, and it was hampering productivity. He looked at it all, and redid almost all of it in something like Frankly…

what's your point exactly? what do you hope to achieve by "bringing it up" (I assume in your workplace)? most programmers are not able to solve a problem like that in 20 lines of assembly or whatever, and no amount of education or awareness is going to change that. acting as if they can is just going to come across as arrogant.

> acting as if they can is just going to come across as arrogant.

Hold on there a sec: WHAT?!

Engineers tend to solve their problems differently and the circumstances for those differences are not always clear. I'm in this field because I want to learn as many different approaches as possible. Did you never experience a moment when you could share a simpler solution to a problem with someone and could observe first hand when they became one of todays lucky 10'000[0]? That's anything but arrogant in my book.

Sadly, what I can increasingly observe is the complete opposite. Nobody wants to talk about their solutions, everyone wants to gatekeep and become indispensable, and criticism isn't seen as part of productive environments as "we just need to ship that damn feature!". Team members should be aware when decisions have been made out of lazyness, in good faith, out of experience, under pressure etc.

[0]: https://xkcd.com/1053/

Re: We were wrong about GPUs

#343

Post by Sparkyte »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Kubernetes is an abstraction of VMs so that single container can be implemented in the absence of a code package. The container is the binary in this circumstance. Unfortunately they lose control of blame shifting if their deployment fails. I can no longer be the VMs fault for failure. What is deployed in lower environments is what is in Prod physically identical outside of configuration.

Re: We were wrong about GPUs

#344

Post by globular-toast »

BobbyTables2 wrote: post #179
nicoburns wrote: post #112

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

Well, the point is you don't have to understand it all at the same time. Kubernetes really just codifies concepts that people were doing before. And it sits on the same foundations (Linux, IP, DNS etc). People writing apps didn't understand the whole thing before, just as they don't now. But at some level these boxes are plugged into each other. A bad system would be one where people writing business software have to care about what box is plugged into what. That's absolutely not the case with Kubernetes.

Re: We were wrong about GPUs

#345

Post by chousuke »

BobbyTables2 wrote: post #179
nicoburns wrote: post #112

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

I think "core" kubernetes is actually pretty easy to understand. You have the kubelet, which just cares about getting pods running, which it does by using pretty standard container tech. You bootstrap a cluster by reading the specs for the cluster control plane pods from disk, after which the kubelet will start polling the API it just started for more of the same. The control plane then takes care of scheduling more pods to the kubelets that have joined the cluster. Pods can run controllers that watch the API for other kinds of resources, but one way or another, most of those get eventually turned into Pod specs that get assigned to a kubelet to run.

Cluster networking can sometimes get pretty mind-bending, but honestly that's true of just containers on their own.

I think just that ability to schedule pods on its own requires about that level of complexity; you're not going to get a much simpler system if you try to implement things yourself. Most of the complexity in k8s comes from components layered on top of that core, but then again, once you start adding features, any custom solution will also grow more complex.

If there's one legitimate complaint when it comes to k8s complexity, it's the ad-hoc way annotations get used to control behaviour in a way that isn't discoverable or type-checked like API objects are, and you just have to be aware that they could exist and affect how things behave. A huge benefit of k8s for me is its built-in discoverability, and annotations hurt that quite a bit.

Re: We were wrong about GPUs

#346

Post by motorest »

jchw wrote: post #298
nicoburns wrote: post #112

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

If you have a system that's actually big or complex enough to warrant using Kubernetes, which, to be frank, isn't really that much considering the realities of production, the only thing more complex than Kubernetes is implementing the same concepts but half-assed. I really wonder why this opinion is so commonly accepted by everyone. I get that not everything needs most Kubernetes features, but it's useful. The Linux…

> If you have a system that's actually big or complex enough to warrant using Kubernetes (...)

I think there's a degree of confusion over your understanding of what Kubernetes is.

Kubernetes is a platform to run containerized applications. Originally it started as a way to simplify the work of putting together clusters of COTS hardware, but since then its popularity drove it to become the platform instead of an abstraction over other platforms.

What this means is that Kubernetes is now a standard way to deploy cloud applications, regardless of complexity or scale. Kubernetes is used to deploy apps to raspberry pis, one-box systems running under your desk, your own workstation, one or more VMs running on random cloud providers, and AWS. That's it.

Re: We were wrong about GPUs

#347

Post by ignoramous »

transpute wrote: post #187
tptacek wrote: post #110

Earlier quoted context omitted.

I assume anything we did to make MIG work for us would have been custom.

Going back to the blog post: > Alternatively, we could have used a conventional hypervisor. Nvidia suggested VMware (heh). But they could have gotten things working had we used QEMU. We like QEMU fine, and could have talked ourselves into a security story for it, but the whole point of Fly Machines is that they take milliseconds to start. Someone could implement virtio-cuda (there are PoCs on github [1] [2]), but it…

> Someone could implement virtio-cuda (there are PoCs on github [1][2]

Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C?

> how to reuse Nvidia's proprietary GPGPU emulation code from QEMU

If it has been contributed to QEMU, it isn't GPL/LGPL?

> Could an emulated vGPGPU be hotplugged after VM launch

gVisor instead bounces ioctls back and forth between "guest" and host. Sounds like a nice, lightweight (even if limited & sandbox-busting) approach, too. Unsure if it mitigates the need for the "licensing dance" tptacek mentioned above, but I reckon the security posture of such a setup is unacceptable for Fly.

https://gvisor.dev/docs/user_guide/gpu/

> would also advance the state of the art in security isolation and access control of emulated devices used by VMs

I hope I'm not talking to DeepSeek / DeepResearch (:

Re: We were wrong about GPUs

#348

Post by vaylian »

sgarland wrote: post #184
SkyPuncher wrote: post #94

Earlier quoted context omitted.

I enjoy the details, but I don’t get paid to tell my executives how we’re running things. I get paid to ship customer facing value. Particularly at startups, it’s almost always more cost effective to hit that “scale up” button from our hosting provider than do any sort of actual system engineering. Eventually, someone goes “hey we could save $$$$ by doing XYZ” so we send someone on a systems engineering journey for a…

Wild idea: maybe if more devs had good fundamental knowledge to begin with, the good systems engineering could be done along the way.

I'm a big fan of fundamental knowledge, but I disagree somewhat with your statement. The thing that startups care most about is a product market fit. And finding that fit requires a lot of iteration and throw-away code. Once the dust settles and you have an initial user base, you can start looking into optimizations.

Re: We were wrong about GPUs

#349

Post by jltsiren »

lifeisstillgood wrote: post #96
Yoric wrote: post #87

Earlier quoted context omitted.

Plenty of non-IT applications use lots of cores, e.g. physics simulations, constraint solving, network simulation used to plan roads or electrical distribution, etc.

Yes - but the amount of code that loves Amdahls law, is tiny compared to amount code churned out each day that can never run parallel over 1M cores - no matter how clever a compiler gets. I cannot work out if we pack enough parallel problems in the world or just lack a programming language to describe them

You can easily use a million cores if you have a million independent tasks. Which you often have, if you are dealing with data rather than building a big centralized service.

Re: We were wrong about GPUs

#350

Post by te_chris »

t0mas88 wrote: post #312
lifeisstillgood wrote: post #307

Earlier quoted context omitted.

Edit: it’s worth expanding on the data centre costs issues - if it’s fair to say we have stalled on “free”speed ups for standard software (ie C/Linux) - that is clock speeds more or less stopped and we get more 64 bit cores but each core is more or less no faster (hand wavy), then the amount of clients that can be packed into a data centre stays the same - you are renting out CPUs in a Docker VM - that’s basically on…

Web applications and APIs for mobile apps are embarrassingly parallel, but many modern web languages and frameworks went back to the stone age on parallelism. Ancient Java servlets back in the early 2000s were more suitable for performance on current gen hardware than modern NodeJS, Python etc...

There’s always Elixir!
Post reply on HN