71–80 of 117 posts
Reminds me of Plan9 :)
can you elaborate a bit on why? (noob here)
So won’t that make the network the prohibitive bottle neck? Your memory bandwidth is 1gbps max
Earlier quoted context omitted.
Great point, there are a few benefits: 1. If you're actively developing and need a GPU then you typically would be paying the entire time the instance is running. Using Thunder means you only pay for the GPU while actively using it. Essentially, if you are running CPU only code you would not be paying for any GPU time. The alterative for this is to manually turn the instance on and off which can be annoying. 2. This…
Okay, but your GPUs are in ECS. Don't I just want this feature from Amazon, not you, and natively via Nitro? Or even Google has TPU attachments. > 1. If you're actively developing and need a GPU [for fractional amounts of time]... Why would I need a GPU for a short amount of time during development? For testing? I don't get it - what would testing an H100 over a TCP connection tell me? It's like, yeah, I can do that,…
This is interesting, but I'm more interested in self-hosting. I already have a lot of GPUs (some running some not.) Does this have a self-hosting option so I can use the GPUs I already have?
[flagged]
Earlier quoted context omitted.
How do you do that exactly? Are you using eBPF or something else? Also, for my ML workloads the most common bottleneck is GPU VRAM RAM copies. Doesn't this dramatically increase latency? Or is it more like it increases latency on first data transfer, but as long as you dump everything into VRAM all at once at the beginning you're fine? I'd expect this wouldn't play super well with stuff like PyTorch data loaders, but…
We intercept api calls and use our own implementation to forward them to a remote machine. No eBPF (which I believe need to run in the kernel). As for latency, we've done a lot of work to minimize that as much as possible. You can see the performance we get running inference on BERT from huggingface here: https://youtu.be/qsOBFQZtsFM?t=64 . It's still slower than local (mainly for training workloads) but not by as mu…
I would would consider using a larger model for demonstrating inference performance as I have 7B models deployed on CPU at work, but GPU is still important training BERT size models.
I'm confused, if this operates at the CPU/GPU boundary doesn't it create a massive I/O bottleneck for any dataset that doesn't fit into VRAM? I'm probably misunderstanding how it works but if it intercepts GPU i/o then it must stream your entire dataset on every epoch to a remote machine, which sounds wasteful, probably I'm not getting this right.
That understanding of the system is correct. To make it practical we've implemented a bunch of optimizations to minimize I/O cost. You can see how it performs on inference with BERT here: https://youtu.be/qsOBFQZtsFM?t=69 . The overheads are larger for training compared to inference, and we are implementing more optimizations to approach native performance.
The same way one "approaches the sun" when they take the stairs?
Ah this is quite interesting! I had a usecase where I needed a GPU-over-IP but only for transcoding videos. I had a not-so-powerful AMD GPU in my homelab server that somehow kept crashing the kernel any time I tried to encode videos with it and also an NVIDIA RTX 3080 in a gaming machine. So I wrote https://github.com/steelbrain/ffmpeg-over-ip and had the server running in the windows machine and the client in the me…
This is more or less what I was hoping for when I saw the submission title. Was disappointed to see that the submission wasn't actually a useful generic tool but instead a paid cloud service. Of course the real content is in the comments. As an aside, are there any uses for GPU-over-network other than video encoding? The increased latency seems like it would prohibit anything machine learning related or graphics inte…
I found Juice to work decently for graphical applications too (e.g., games, CAD software). Latency was about what you'd expect for video encode + decode + network: 5-20ms on a LAN if I recall correctly.
For anyone curious about how this actually works, it looks like a library is injected into your process to hook these functions [1] in order to forward them to the service. [1] https://pastebin.com/raw/kCYmXr5A
I'm confused, if this operates at the CPU/GPU boundary doesn't it create a massive I/O bottleneck for any dataset that doesn't fit into VRAM? I'm probably misunderstanding how it works but if it intercepts GPU i/o then it must stream your entire dataset on every epoch to a remote machine, which sounds wasteful, probably I'm not getting this right.
That understanding of the system is correct. To make it practical we've implemented a bunch of optimizations to minimize I/O cost. You can see how it performs on inference with BERT here: https://youtu.be/qsOBFQZtsFM?t=69 . The overheads are larger for training compared to inference, and we are implementing more optimizations to approach native performance.