Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Attaching to a virtual GPU over TCP

thundercompute.com

Post reply on HN
Print view View original

21–30 of 117 posts

Re: Show HN: Attaching to a virtual GPU over TCP

#21

Post by tptacek »

bmodel wrote: post #17
tptacek wrote: post #15

This is neat. Were you able to get MIG or vGPUs working with it?

We haven't tested with MIG or vGPU, but I think it would work since it's essentially physically partitioning the GPU. One of our main goals for the near future is to allow GPU sharing. This would be better than MIG or vGPU since we'd allow users to use the entire GPU memory instead of restricting them to a fraction.

We had a hell of a time dealing with the licensing issues and ultimately just gave up and give people whole GPUs.

What are you doing to reset the GPU to clean state after a run? It's surprisingly complicated to do this securely (we're writing up a back-to-back sequence of audits we did with Atredis and Tetrel; should be publishing in a month or two).

Re: Show HN: Attaching to a virtual GPU over TCP

#23

Post by bmodel »

orsorna wrote: post #19

So what exactly is the pricing model? Do I need a quote? Because otherwise I don't see how to determine it without creating an account which is needlessly gatekeeping.

We're still in our beta so it's entirely free for now (we can't promise a bug-free experience)! You have to make an account but it won't require payment details.

Down the line we want to move to a pay-as-you-go model.

Re: Show HN: Attaching to a virtual GPU over TCP

#29

Post by bmodel »

ZeroCool2u wrote: post #20
bmodel wrote: post #8

Earlier quoted context omitted.

We are not writing any kernel drivers, this runs entirely in userspace (this won't result in a crowdstrike level crash haha). Given that, if the network suddenly dropped then only the process using the GPU would fail.

How do you do that exactly? Are you using eBPF or something else? Also, for my ML workloads the most common bottleneck is GPU VRAM RAM copies. Doesn't this dramatically increase latency? Or is it more like it increases latency on first data transfer, but as long as you dump everything into VRAM all at once at the beginning you're fine? I'd expect this wouldn't play super well with stuff like PyTorch data loaders, but…

We intercept api calls and use our own implementation to forward them to a remote machine. No eBPF (which I believe need to run in the kernel).

As for latency, we've done a lot of work to minimize that as much as possible. You can see the performance we get running inference on BERT from huggingface here: https://youtu.be/qsOBFQZtsFM?t=64. It's still slower than local (mainly for training workloads) but not by as much as you'd expect. We're aiming to reach near parity in the next few months!

Re: Show HN: Attaching to a virtual GPU over TCP

#30

Post by bmodel »

rubatuga wrote: post #27

What ML packages do you support? In the comments below it says you do not support Vulkan or OpenGL. Does this support AMD GPUs as well?

We have tested this with pytorch and huggingface and it is mostly stable (we know there are issues with pycuda and jax). In theory this should work with any libraries, however we're still actively developing this so bugs will show up
Post reply on HN