Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Attaching to a virtual GPU over TCP

thundercompute.com

Post reply on HN
Print view View original

51–60 of 117 posts

Re: Show HN: Attaching to a virtual GPU over TCP

#51

Post by bmodel »

radarsat1 wrote: post #46

I'm confused, if this operates at the CPU/GPU boundary doesn't it create a massive I/O bottleneck for any dataset that doesn't fit into VRAM? I'm probably misunderstanding how it works but if it intercepts GPU i/o then it must stream your entire dataset on every epoch to a remote machine, which sounds wasteful, probably I'm not getting this right.

That understanding of the system is correct. To make it practical we've implemented a bunch of optimizations to minimize I/O cost. You can see how it performs on inference with BERT here: https://youtu.be/qsOBFQZtsFM?t=69.

The overheads are larger for training compared to inference, and we are implementing more optimizations to approach native performance.

Re: Show HN: Attaching to a virtual GPU over TCP

#52

Post by goku-goku »

bmodel wrote: post #16
billconan wrote: post #12

Earlier quoted context omitted.

it's more transparent to your system, for example, if you have a gui application that needs gpu acceleration on a thin client (Matlab, solidworks, blender), you can do so without setting up ECS. you can develop without any gpu, but suddenly have one when you need to run simulation. this will be way cheaper than AWS. I think essentially this is solving the same problem Ray ( https://www.ray.io/ ) is solving, but in a…

Exactly! The finer grain sharing is one of the key things on our radar right now

www.juicelabs.co does all this today, including the GPU sharing and fractionalization.

Re: Show HN: Attaching to a virtual GPU over TCP

#56

Post by samstave »

bmodel wrote: post #29
ZeroCool2u wrote: post #20

Earlier quoted context omitted.

How do you do that exactly? Are you using eBPF or something else? Also, for my ML workloads the most common bottleneck is GPU VRAM RAM copies. Doesn't this dramatically increase latency? Or is it more like it increases latency on first data transfer, but as long as you dump everything into VRAM all at once at the beginning you're fine? I'd expect this wouldn't play super well with stuff like PyTorch data loaders, but…

We intercept api calls and use our own implementation to forward them to a remote machine. No eBPF (which I believe need to run in the kernel). As for latency, we've done a lot of work to minimize that as much as possible. You can see the performance we get running inference on BERT from huggingface here: https://youtu.be/qsOBFQZtsFM?t=64 . It's still slower than local (mainly for training workloads) but not by as mu…

When you release a self-host version, what would be really neat would be to see it across HFT focused NICs that have huge TCP buffers...

https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetwork...

Basically taking into account the large buffers and super-time-sensitive nature of HFT networking optimizations, I wonder if your TCPGPU might benefit from both the HW and the learnings of NFT stylings?

Re: Show HN: Attaching to a virtual GPU over TCP

#58

Post by radarsat1 »

bmodel wrote: post #51
radarsat1 wrote: post #46

I'm confused, if this operates at the CPU/GPU boundary doesn't it create a massive I/O bottleneck for any dataset that doesn't fit into VRAM? I'm probably misunderstanding how it works but if it intercepts GPU i/o then it must stream your entire dataset on every epoch to a remote machine, which sounds wasteful, probably I'm not getting this right.

That understanding of the system is correct. To make it practical we've implemented a bunch of optimizations to minimize I/O cost. You can see how it performs on inference with BERT here: https://youtu.be/qsOBFQZtsFM?t=69 . The overheads are larger for training compared to inference, and we are implementing more optimizations to approach native performance.

Aah ok thanks, that was my basic misunderstanding, my mind just jumped straight to my current training needs but for inference it makes a lot of sense. Thanks for the clarification.

Re: Show HN: Attaching to a virtual GPU over TCP

#60

Post by test20240809 »

pocl (Portable Computing Language) [1] provides a remote backend [2] that allows for serialization and forwarding of OpenCL commands over a network.

Another solution is qCUDA [3] which is more specialized towards CUDA.

In addition to these solutions, various virtualization solutions today provide some sort of serialization mechanism for GPU commands, so they can be transferred to another host (or process). [4]

One example is the QEMU-based Android Emulator. It is using special translator libraries and a "QEMU Pipe" to efficiently communicate GPU commands from the virtualized Android OS to the host OS [5].

The new Cuttlefish Android emulator [6] uses Gallium3D for transport and the virglrenderer library [7].

I'd expect that the current virtio-gpu implementation in QEMU [8] might make this job even easier, because it includes the Android's gfxstream [9] (formerly called "Vulkan Cereal") that should already support communication over network sockets out of the box.

[1] https://github.com/pocl/pocl

[2] https://portablecl.org/docs/html/remote.html

[3] https://github.com/coldfunction/qCUDA

[4] https://www.linaro.org/blog/a-closer-look-at-virtio-and-gpu-...

[5] https://android.googlesource.com/platform/external/qemu/+/em...

[6] https://source.android.com/docs/devices/cuttlefish/gpu

[7] https://cs.android.com/android/platform/superproject/main/+/...

[8] https://www.qemu.org/docs/master/system/devices/virtio-gpu.h...

[9] https://android.googlesource.com/platform/hardware/google/gf...

Post reply on HN