I shelled out for a 4090 when they came out thinking it would be the key factor for running local llms. It turns out that anything worth running takes way more than 24GB VRAM. I would have been better off with 2+ 3090s and a custom power supply. It’s a pity because I thought it would be a great solution for coding and a home assistant, but performance and quality isn’t there yet for small models (afaik). Perhaps DIGITS will scratch the itch for local LLM developers, but performant models really want big metal for now, not something I can afford to own or rent at my scale.
We were wrong about GPUs
41–50 of 604 posts
Re: We were wrong about GPUs
#42Side note: "we were wrong" - are there any more noble and beautiful words in the English language?
I don't know.
Re: We were wrong about GPUs
#43I really liked playing around with fly gpus, but it's just too expensive for hobby-use. Same goes for the rest of fly.io honestly. The DX is great and I wish I could move all of my homelab stuff and public websites to it, but it'd be way too expensive :(
This is near and dear to me, because I want people to run stuff like homelabs and side projects.
What part of the cost gets out of hand? Having to have a Machine for every process? Do you remember what napkin math pricing you were working with?
Re: We were wrong about GPUs
#44I noticed quite a few spelling and grammar mistakes - could do with a bit of an edit pass?
In the current days of AI I think spelling and grammar mistakes is perhaps a great way to tell it is still written by human...... ( Until AI copy this )
I do, in fact, instruct LLMs to make spelling and grammar mistakes when I have them reply to cold emails.
Re: We were wrong about GPUs
#45Earlier quoted context omitted.
> We burned months trying (and ultimately failing) to get Nvidia’s host drivers working to map virtualized GPUs into Intel Cloud Hypervisor... We think there’s probably a market for users doing lightweight ML work getting tiny GPUs. This is what Nvidia MIG does, slicing a big GPU into arbitrarily small virtual GPUs. But for fully-virtualized workloads, it’s not baked; we can’t use it. Near as we can tell, MIG gives y…
We (I) spent a lot of time talking to several different teams at Nvidia about this. We were able to get VFIO vGPUs to the point where guest libraries would recognize them, but the process fell apart in the guest/host licensing dance, and we weren't really OK with the idea that there'd be a phone-home licensing dance every time a Fly Machine started. Unlike GPU enablement at GCP or AWS, the core DX of a Fly Machine is…
> phone-home licensing dance every time a Fly Machine started
To userspace Nvidia license server (a) in each host, (b) for entire Fly cloud, or (c) over WAN to Nvidia cloud?
Re: We were wrong about GPUs
#46I shelled out for a 4090 when they came out thinking it would be the key factor for running local llms. It turns out that anything worth running takes way more than 24GB VRAM. I would have been better off with 2+ 3090s and a custom power supply. It’s a pity because I thought it would be a great solution for coding and a home assistant, but performance and quality isn’t there yet for small models (afaik). Perhaps DIGI…
There was a post on r/localLlama the other day about a presentation by the company building Digits hardware for Nvidia. The gist was that Digits is going to be aimed at academic AI research folks and as such don't expect them to be available in large numbers (at least not for this first version). It was disappointing. Now I'm awaiting the AMD Strix Halo based systems.
Re: We were wrong about GPUs
#47Earlier quoted context omitted.
We (I) spent a lot of time talking to several different teams at Nvidia about this. We were able to get VFIO vGPUs to the point where guest libraries would recognize them, but the process fell apart in the guest/host licensing dance, and we weren't really OK with the idea that there'd be a phone-home licensing dance every time a Fly Machine started. Unlike GPU enablement at GCP or AWS, the core DX of a Fly Machine is…
> phone-home licensing dance every time a Fly Machine started To userspace Nvidia license server (a) in each host, (b) for entire Fly cloud, or (c) over WAN to Nvidia cloud?
IIRC, (a) and (c), which I think sort of implies (b)?
Really what we'd have wanted to do would have been to give Fly Machines MIG slices. But to the best of my understanding, MIG is paravirtualized; it doesn't give you SR-IOV-style PCI addresses for the slices, but rather a reference Nvidia's userland libraries pass to the kernel driver, which is a dance you can't do across VM boundaries unless your hypervisor does it deliberately.
Re: We were wrong about GPUs
#48I'm admittedly a complete LLM noob, so my question might not even make sense. Or it might exist and I haven't found it quite yet.
But have they considered pivoting some of said compute to some 'private, secure LLM in a box' solution?
I've lately been toying with the idea of training from extensive docs and code, some open, some not, for both code generation and insights.
I went down the RAG rabbit hole, and frankly, the amount of competing ideas of 'this is how you should do it', from personal blogs to PaaS companies, overwhelmed me. Vector dbs, ollama, models, langchain, and various one off tools linking to git repos.
I feel there has to be substantial market for whoever can completely simplify that flow for dummies like me, and not charge a fortune for the privilege.
Re: We were wrong about GPUs
#49I noticed quite a few spelling and grammar mistakes - could do with a bit of an edit pass?
Nah, fly.io has a company culture that is all about having lots of bugs and issues, and that includes blog posts. The idea that a cloud compute provider can’t make GPU compute into an profitable business is pretty laughable.
For what it’s worth I don’t think we entirely disagree: it has at times felt absurd that it didn’t make as much money as it maybe otherwise could. We made a bet that the type of cloud platform we wanted to build could be well served by GPUs. It wasn’t as good a bet as we thought. There is probably a different type of cloud product we could build that would be better set up to sell gpus but we are still committed to the primitives our machine product has to offer.
Re: We were wrong about GPUs
#50Out of curiosity, how much runway does fly.io have (without raising new funding?)