Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our container platform is in production. It has GPUs. Here's an early look

blog.cloudflare.com

Post reply on HN
Print view View original

41–50 of 76 posts

Re: Our container platform is in production. It has GPUs. Here's an early look

#41

Post by bigcat12345678 »

dopylitty wrote: post #37

I like the dig at "first generation" clouds. There really is a wide gulf between the services provided by the older cloud providers (AWS, Azure) and the newer ones (fly.io, CloudFlare etc). AWS/Azure provide very leaky abstractions (VMs, VPCs) on top of very old and badly designed protocols/systems (IP, Windows, Linux) . That's fine for people who want to spend all their time janitoring VMs, operating systems, and ne…

Hn now clearly are swarmed by grandiose novice techs.

10 years ago, no such superficial assessment would appear on first page.

This set of words bear little substance and engineering facts.

> AWS/Azure provide very leaky abstractions (VMs, VPCs) on top of very old and badly designed protocols/systems (IP, Windows, Linux) .

AWS cannot be made parallel, they themselves are 2 gens

AWS gen1

Azure gcp gen 2

Gen1 is on vm, ecs ebs s3, for web2 era

Gen2 is on cluster computing which was enable by vm

The then "leaky abstraction" is the mandated abstraction at the time

And GPUs today is about 70s's CPU

For example, you don't have any form of abstracted runtime on GPU, it's like running dos system

It's more leaky than 00s ' vm

Re: Our container platform is in production. It has GPUs. Here's an early look

#42

Post by mnahkies »

wmf wrote: post #19
tomrod wrote: post #8

This seems like a pretty big deal. I want to like CloudFlare over DO/AWS. I like their DevX focus too -- I could see issues if devs can't get into the abstractions though. Any red flags folks would stake regarding CF? I know they are widely used but not sure where the gotchas are.

Is Cloudflare the one that goes from free to "call for pricing" ($100K+) at the drop of a hat?

I think they have some incredibly low pricing for what most small companies need. I also think they've done a very good job of carving out pieces that more sophisticated setups need into the enterprise tier, which does constitute a big jump.

One that bit me was https://developers.cloudflare.com/cloudflare-for-platforms/c... - we found an alternative solution that didn't require upgrading to their enterprise plan (yet), but it was a pretty compelling reason to upgrade and if I was doing it again I'd probably choose upgrading over implementing our solution. On balance I'm not sure we actually saved money in the end, considering opportunity cost

Re: Our container platform is in production. It has GPUs. Here's an early look

#44

Post by ec109685 »

ckastner wrote: post #16
ec109685 wrote: post #10

Earlier quoted context omitted.

If the calls first pass through a memory safe language as what gvisor does, isn’t the attack surface greatly reduced? It does seem however that Firecracker + GPU support (or https://github.com/cloud-hypervisor/cloud-hypervisor ) is most promising though. It’s surprising that AWS doesn’t have a need for Lambda but with GPU’s to motivate them to bring GPU’s to firecracker.

> If the calls first pass through a memory safe language as what gvisor does, isn’t the attack surface greatly reduced? The runtime may be memory safe, but I'm thinking of the GPU workloads which nvproxy seems to pass on to the device via the host's kernel. Say I find a security issue in the GPU's driver, and manage to exploit it with some malicious CUDA workload.

Would having a VM inbetween help in that case? It seems like protecting against malicious GPU workloads requires the GPU to off virtualization to avoid this exploit.

This is helpful in explaining why AWS hasn't been excited to ship this use case in firecracker.

Re: Our container platform is in production. It has GPUs. Here's an early look

#45

Post by ec109685 »

hinkley wrote: post #22
ec109685 wrote: post #9

Earlier quoted context omitted.

Their solution isn’t GA yet. For headless browsers, the latency benefits of “container anywhere” seems high. For things like AI inference, running on the edge seems way less beneficial than running on the cheapest location possible which would be larger regional data centers.

One would hope that “larger regional data centers” are not that far from The Edge. But the problem isn’t physics or the speed of light, it’s operational. The operational excellence required to have every successful Internet company manage deployments to a dozen regions just isn’t there. Most of us struggle with three, my last gig tried to do two, which isn’t economical because you always try to handle one region goin…

Given how slow AI inference is (and for training it doesn't matter at all), the advantage of it being a few milliseconds closer to the user is greatly diminished. The latency to egress to a regional data center is inconsequential.

Good point about at the very least not exposing placement to customers. That is a definite win.

Re: Our container platform is in production. It has GPUs. Here's an early look

#47

Post by tscolari »

roboben wrote: post #46

What I am always missing in these posts: How do they limit network bandwidth? Since these are all multi-tenant services, how do they make sure a container or isolated browser is not taking all the network bandwidth of a host?

You probably can do this through the proc filesystem/cgroups. If you think about it, you can use cgroups to limit the bandwidth, so you can also use it to measure it.

Re: Our container platform is in production. It has GPUs. Here's an early look

#49

Post by hinkley »

ckastner wrote: post #25
hinkley wrote: post #20

Earlier quoted context omitted.

I’ve been looking at distributed CI and for now I’m just going to be running workloads queued by the owner of the agent. That doesn’t eliminate hostile workloads but it does present a similar surface area to simply running the builds locally. I’ve been thinking about QEMM or firecracker instead of just containers for a more robust solution. I have some time before anyone would ask me about GPU workloads, but do you t…

Amazon/AWS has no use case for VFIO in Firecracker. They're open to the community adding support and have a community meeting soon, but I wouldn't get my hopes up. QEMU can work -- I say can, because it doesn't work with all GPUs. And with consumer GPUs, VFIO is generally not an officially supported use case. We got it working, but with lots of trial and error, and there are still some problematic corner cases.

What would you say is the sort of time horizon for turnkey operation of one commonly available video card, half a dozen, and OEM cards in high end laptops (eg, MacBook Pro)? Years? Decades? Heat death?

Re: Our container platform is in production. It has GPUs. Here's an early look

#50

Post by trallnag »

ignoramous wrote: post #38
wmf wrote: post #19

Earlier quoted context omitted.

Is Cloudflare the one that goes from free to "call for pricing" ($100K+) at the drop of a hat?

One data point but, one among our toy services has been pushing 30TB/mo to 60TB/mo for over a year now, and we haven't got the call: https://news.ycombinator.com/item?id=39521228

Can you share what kind of toy is shuffling around so much data?
Post reply on HN