Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

21–30 of 604 posts

Re: We were wrong about GPUs

#21

Post by tptacek »

akoculu wrote: post #18
tptacek wrote: post #13

Earlier quoted context omitted.

I mean, yes? Managing giant model weight files is a big problem with getting people on-demand access to Docker-based micro-VMs. I don't think we missed that point so much as that we acknowledged it, and found some clarity in the idea that we weren't going to break up our existing DX just to fix it. If there were lots and lots and lots of people trying to self-host LLMs running into this problem, it would have been a…

Did you consider other use cases in which people need custom models and inference other than just open source LLMs ?

Yes. Click through to the L40S post the article links to (the L40S's aren't going anywhere).

There are people doing GPU-enabled inference stuff on Fly.io. That particular slice of the market seems fine?

Re: We were wrong about GPUs

#22

Post by onli »

Not sure about this:

> like with our portfolio of IPv4 addresses, I’m even more comfortable making bets backed by tradable assets with durable value.

Is that referencing the gpus, the hardware? If yes, why should they have a durable value? Historically hardware like that deprecated fast and reaches a value of 0, energy efficiency alone kills e.g. old server hardware. Something different here?

Re: We were wrong about GPUs

#23

Post by _zoltan_ »

dangus wrote: post #16
Philpax wrote: post #11

I noticed quite a few spelling and grammar mistakes - could do with a bit of an edit pass?

Nah, fly.io has a company culture that is all about having lots of bugs and issues, and that includes blog posts. The idea that a cloud compute provider can’t make GPU compute into an profitable business is pretty laughable.

I have to agree with this. Look at GPU utilization at AWS, Azure, .. they are running close to 100%.

for our p5 quota I had to talk to our TAM team on AWS, while most of our quota requests are instant usually.

Re: We were wrong about GPUs

#24

Post by transpute »

tptacek wrote: post #2

We wrote all sorts of stuff this week and this is what gets to the front page. :P

> We burned months trying (and ultimately failing) to get Nvidia’s host drivers working to map virtualized GPUs into Intel Cloud Hypervisor... We think there’s probably a market for users doing lightweight ML work getting tiny GPUs. This is what Nvidia MIG does, slicing a big GPU into arbitrarily small virtual GPUs. But for fully-virtualized workloads, it’s not baked; we can’t use it. Near as we can tell, MIG gives you a UUID to talk to the host driver, not a PCI device.

Apparently this is technically possible, if you can find the right person at Nvidia to talk about vGPU licensing and magic incantations. Hopefully someone reading this HN front page story can make the introduction.

Re: We were wrong about GPUs

#25

Post by serjester »

I respect them for being public about this.

With that said, this seems quite obvious - the type of customer that chooses Fly, seems like the last person to be spinning up dedicated GPU servers for extended periods of time. Seems much more likely they'll use something serverless which requires a ton of DX work to get right (personally I think Modal is killing it here). To compete, they would have needed to bet the company on it. It's way too competitive otherwise.

Re: We were wrong about GPUs

#26

Post by johntash »

I really liked playing around with fly gpus, but it's just too expensive for hobby-use. Same goes for the rest of fly.io honestly. The DX is great and I wish I could move all of my homelab stuff and public websites to it, but it'd be way too expensive :(

Re: We were wrong about GPUs

#28

Post by tptacek »

transpute wrote: post #24
tptacek wrote: post #2

We wrote all sorts of stuff this week and this is what gets to the front page. :P

> We burned months trying (and ultimately failing) to get Nvidia’s host drivers working to map virtualized GPUs into Intel Cloud Hypervisor... We think there’s probably a market for users doing lightweight ML work getting tiny GPUs. This is what Nvidia MIG does, slicing a big GPU into arbitrarily small virtual GPUs. But for fully-virtualized workloads, it’s not baked; we can’t use it. Near as we can tell, MIG gives y…

We (I) spent a lot of time talking to several different teams at Nvidia about this. We were able to get VFIO vGPUs to the point where guest libraries would recognize them, but the process fell apart in the guest/host licensing dance, and we weren't really OK with the idea that there'd be a phone-home licensing dance every time a Fly Machine started. Unlike GPU enablement at GCP or AWS, the core DX of a Fly Machine is that it stop/start very quickly; think of it as a midpoint in the design space between Lambda and Fargate. This is what we're talking about when we say it's hard to fit GPUs into our DX.

Re: We were wrong about GPUs

#30

Post by johntash »

jeffybefffy519 wrote: post #9

I feel like these guys are missing a pretty important point in their own analysis. I tried setting up a ollama LLM on a fly.io GPU machine and it was near impossible because of fly.io limitations such as: 1. Their infrastructure doesnt support streaming responses well at all (which is important part of the LLM experience in my view) 2. The LLM itself is massive, and cant be part of the docker image I was building and…

What kind of issues did you have with streaming? I also set up ollama on fly.io, and had no issues getting streaming to work.

For the LLM itself, I just used a custom startup script that downloaded the model once ollama was up. It's the same thing I'd do on a local cluster though. I'm not sure how fly could make it better unless they offered direct integration with ollama or some other inference server?

Post reply on HN