Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Attaching to a virtual GPU over TCP

thundercompute.com

Post reply on HN
Print view View original

11–20 of 117 posts

Re: Show HN: Attaching to a virtual GPU over TCP

#11

Post by steelbrain »

Ah this is quite interesting! I had a usecase where I needed a GPU-over-IP but only for transcoding videos. I had a not-so-powerful AMD GPU in my homelab server that somehow kept crashing the kernel any time I tried to encode videos with it and also an NVIDIA RTX 3080 in a gaming machine.

So I wrote https://github.com/steelbrain/ffmpeg-over-ip and had the server running in the windows machine and the client in the media server (could be plex, emby, jellyfin etc) and it worked flawlessly.

Re: Show HN: Attaching to a virtual GPU over TCP

#12

Post by billconan »

doctorpangloss wrote: post #10

I don't get it. Why would I start an instance in ECS, to use your GPUs in ECS, when I could start an instance for the GPUs I want in ECS? Separately, why would I want half of Nitro, instead of real Nitro?

it's more transparent to your system, for example, if you have a gui application that needs gpu acceleration on a thin client (Matlab, solidworks, blender), you can do so without setting up ECS. you can develop without any gpu, but suddenly have one when you need to run simulation. this will be way cheaper than AWS.

I think essentially this is solving the same problem Ray (https://www.ray.io/) is solving, but in a more generic way.

it potentially can have finer grained gpu sharing, like a half-gpu.

I'm very excited about this.

Re: Show HN: Attaching to a virtual GPU over TCP

#13

Post by bmodel »

doctorpangloss wrote: post #10

I don't get it. Why would I start an instance in ECS, to use your GPUs in ECS, when I could start an instance for the GPUs I want in ECS? Separately, why would I want half of Nitro, instead of real Nitro?

Great point, there are a few benefits:

1. If you're actively developing and need a GPU then you typically would be paying the entire time the instance is running. Using Thunder means you only pay for the GPU while actively using it. Essentially, if you are running CPU only code you would not be paying for any GPU time. The alterative for this is to manually turn the instance on and off which can be annoying.

2. This allows you to easily scale the type and number of GPUs you're using. For example, say you want to do development on a cheap T4 instance and run a full DL training job on a set of 8 A100. Instead of needing to swap instances and setup everything again, you can just run a command and then start running on the more powerful GPUs.

Re: Show HN: Attaching to a virtual GPU over TCP

#16

Post by bmodel »

billconan wrote: post #12
doctorpangloss wrote: post #10

I don't get it. Why would I start an instance in ECS, to use your GPUs in ECS, when I could start an instance for the GPUs I want in ECS? Separately, why would I want half of Nitro, instead of real Nitro?

it's more transparent to your system, for example, if you have a gui application that needs gpu acceleration on a thin client (Matlab, solidworks, blender), you can do so without setting up ECS. you can develop without any gpu, but suddenly have one when you need to run simulation. this will be way cheaper than AWS. I think essentially this is solving the same problem Ray ( https://www.ray.io/ ) is solving, but in a…

Exactly! The finer grain sharing is one of the key things on our radar right now

Re: Show HN: Attaching to a virtual GPU over TCP

#17

Post by bmodel »

tptacek wrote: post #15

This is neat. Were you able to get MIG or vGPUs working with it?

We haven't tested with MIG or vGPU, but I think it would work since it's essentially physically partitioning the GPU.

One of our main goals for the near future is to allow GPU sharing. This would be better than MIG or vGPU since we'd allow users to use the entire GPU memory instead of restricting them to a fraction.

Re: Show HN: Attaching to a virtual GPU over TCP

#20

Post by ZeroCool2u »

bmodel wrote: post #8
billconan wrote: post #5

Earlier quoted context omitted.

does this mean you have a customized/dummy kernel gpu driver? will that cause system instability, say, if the network suddenly dropped?

We are not writing any kernel drivers, this runs entirely in userspace (this won't result in a crowdstrike level crash haha). Given that, if the network suddenly dropped then only the process using the GPU would fail.

How do you do that exactly? Are you using eBPF or something else?

Also, for my ML workloads the most common bottleneck is GPU VRAM RAM copies. Doesn't this dramatically increase latency? Or is it more like it increases latency on first data transfer, but as long as you dump everything into VRAM all at once at the beginning you're fine? I'd expect this wouldn't play super well with stuff like PyTorch data loaders, but would be curious to hear how you've faired when testing.

Post reply on HN