Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

361–370 of 604 posts

Re: We were wrong about GPUs

#361

Post by transpute »

ignoramous wrote: post #347
transpute wrote: post #187

Earlier quoted context omitted.

Going back to the blog post: > Alternatively, we could have used a conventional hypervisor. Nvidia suggested VMware (heh). But they could have gotten things working had we used QEMU. We like QEMU fine, and could have talked ourselves into a security story for it, but the whole point of Fly Machines is that they take milliseconds to start. Someone could implement virtio-cuda (there are PoCs on github [1] [2]), but it…

> Someone could implement virtio-cuda (there are PoCs on github [1][2] Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C? > how to reuse Nvidia's proprietary GPGPU emulation code from QEMU If it has been contributed to QEMU, it isn't GPL/LGPL? > Could an emulated vGPGPU be hotplugged after VM launch gVisor instead bounces ioctl s back and forth between "guest" and host. Sounds like a nice,…

> Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C?

Good question for a lawyer. Even more reason (beyond maintenance cost) that it would be best done by Nvidia. qCUDA paper has a couple dozen references on API remoting research, https://www.cs.nthu.edu.tw/~ychung/conference/2019-CloudCom....

> If it has been contributed to QEMU, it isn't GPL/LGPL?

Not contributed, but integrated with QEMU by commercial licensees. Since the GPGPU emulation code isn't public, presumably it's a binary blob.

> I hope I'm not talking to DeepSeek / DeepResearch (:

Will take that as a compliment :) Not yet tried DS/DR.

Re: We were wrong about GPUs

#362

Post by sethammons »

mdaniel wrote: post #238
sgarland wrote: post #193

Earlier quoted context omitted.

If you don’t have staff to do that, you probably aren’t at the scale when you need them, and you’re needlessly adding complexity. It’s always baffling to me why people think that ECS or god forbid EKS is somehow easier than a few Linux boxes.

Because the force multiplier of a good DX way outweighs the occasional nonsense from having to do k8s upgrades or troubleshooting For example: how do you roll out a new release of your product? In sane setups, it's often $(helm upgrade --install ...), which is itself often run either in-cluster by watching a git managed descriptor, or in CI on merge to a release branch/tag How does your developer get logs? Maybe it's…

> Splunk/ELK/DataDog/whatever but I have never in my life seen a case where that's a replacement for viewing the logs

Uh, any time I run a distributed system and logs could appear on n nodes I need a log aggregator or I am tailing in n terminals. I almost only use Splunk. I tail logs in dev. Prod needs an aggregator. This has been my experience at 4 of my last 6 companies. The shit companies who had all the issues? Logs on cloudwatch or only on the node

Re: We were wrong about GPUs

#363

Post by [deleted] »

ignoramous wrote: post #347
transpute wrote: post #187

Earlier quoted context omitted.

Going back to the blog post: > Alternatively, we could have used a conventional hypervisor. Nvidia suggested VMware (heh). But they could have gotten things working had we used QEMU. We like QEMU fine, and could have talked ourselves into a security story for it, but the whole point of Fly Machines is that they take milliseconds to start. Someone could implement virtio-cuda (there are PoCs on github [1] [2]), but it…

> Someone could implement virtio-cuda (there are PoCs on github [1][2] Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C? > how to reuse Nvidia's proprietary GPGPU emulation code from QEMU If it has been contributed to QEMU, it isn't GPL/LGPL? > Could an emulated vGPGPU be hotplugged after VM launch gVisor instead bounces ioctl s back and forth between "guest" and host. Sounds like a nice,…

[deleted]

Re: We were wrong about GPUs

#364

Post by bonzini »

transpute wrote: post #361
ignoramous wrote: post #347

Earlier quoted context omitted.

> Someone could implement virtio-cuda (there are PoCs on github [1][2] Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C? > how to reuse Nvidia's proprietary GPGPU emulation code from QEMU If it has been contributed to QEMU, it isn't GPL/LGPL? > Could an emulated vGPGPU be hotplugged after VM launch gVisor instead bounces ioctl s back and forth between "guest" and host. Sounds like a nice,…

> Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C? Good question for a lawyer. Even more reason (beyond maintenance cost) that it would be best done by Nvidia. qCUDA paper has a couple dozen references on API remoting research, https://www.cs.nthu.edu.tw/~ychung/conference/2019-CloudCom.... > If it has been contributed to QEMU, it isn't GPL/LGPL? Not contributed, but integrated with QEMU…

NVIDIA support is not special as far as QEMU is concerned—the special parts are all in their proprietary device driver, and they talk to QEMU via the VFIO infrastructure for userspace drivers. They just reimplemented the same thing in Cloud Hypervisor.

Red Hat for one doesn't ship any functionality that isn't available upstream, much less proprietary, and they have large customers using virtual GPU.

Re: We were wrong about GPUs

#366

Post by ozgune »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

I agree with the blog post that using K8s + containers for GPU virtualization is a security disaster waiting to happen. Even if you configure your container right (which is extremely hard to do), you don't get seccomp-bpf.

People started using K8s for training, where you already had a network isolated cluster. Extending the K8s+container pattern to multi-tenant environments is scary at best.

I didn't understand the following part though.

> Instead, we burned months trying (and ultimately failing) to get Nvidia’s host drivers working to map virtualized GPUs into Intel Cloud Hypervisor.

Why was this part so hard? Doing PCI passthrough with the Cloud Hypervisor (CH) is relatively common. Was it the transition from Firecracker to CH that was tricky?

Re: We were wrong about GPUs

#367

Post by arijo »

motorest wrote: post #319
arijo wrote: post #302

Earlier quoted context omitted.

Great opportunity for someone ballsy to write a book about kubernetes internals for the general engineering population. Bonus points for writing a basic implementation from first principles capturing the essence of the problem kubernetes really was meant to solve. The 100 pages kubernetes book, Andriy Burkov style.

> Great opportunity for someone ballsy to write a book about kubernetes internals for the general engineering population. What would be the interest of it? Think about it: - kubernetes is an interface and not a specific implementation, - the bulk of the industry standardized on managed services, which means you actually have no idea what are the actual internals driving your services, - so you read up on the exact fu…

I don't really care about the standardized interface.

I just want to know how you'd implement something that would load your services and dependencies from a config file, bind them altogether, distribute the load through several local VMs and make it still work if I kill the service or increase the load.

In less than 1000 lines.

Re: We were wrong about GPUs

#368

Post by scosman »

> But inference latency just doesn’t seem to matter yet, so the market doesn’t care.

This is a very strange statement to make. They are acting like inference today happens with freshly spun up VMs and model access over remote networks (and their local switching could save the day). It’s actually hitting clusters of hot machines with the model of choice already loaded into VRAM.

In real deployments, latency can be small (if implemented well), and speed is comes down to the right GPU config for the model (why fly doesn’t offer).

People have built better shared resource usage inference systems for Loras (openAI, Fireworks, Lorax) - but it’s not VMs. It’s model aware, the right hardware for the base model, and optimizing caching/swapping the Loras.

I’m not sure the Fly/VM way will ever be the path for ML. Their VM cold start time doesn’t matter if the app startup requires loading 20GB+ of weights.

Companies like Fireworks are working fast Lora inference cold starts. Companies like Modal are working on fast serverless VM cold starts with a range of GPU configs (2xH100, A100, etc). These seem more like the two cloud primitives for AI.

Re: We were wrong about GPUs

#369

Post by arijo »

laggyluke wrote: post #328
arijo wrote: post #302

Earlier quoted context omitted.

Great opportunity for someone ballsy to write a book about kubernetes internals for the general engineering population. Bonus points for writing a basic implementation from first principles capturing the essence of the problem kubernetes really was meant to solve. The 100 pages kubernetes book, Andriy Burkov style.

You might be interested in this: https://github.com/kelseyhightower/kubernetes-the-hard-way It probably won't answer the "why" (although any LLM can answer that nowadays), but it will definitely answer the "how".

That's nice but I was looking more for a simple implementation of the concept from first principles.

I mean an understanding from the view of the internals and not so much the user perspective.

Post reply on HN