Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

51–60 of 604 posts

Re: We were wrong about GPUs

#51

Post by inetknght »

Kudos to owning up to your failed bet on GPUs even if you are putting a lot of blame on Nvidia for it. And to be fair, you're not wrong. Nvidia's artificial market segmentation is terrible and their drivers aren't that great either.

The real problem is the lack of security-isolated slicing one or more GPUs for virtual machines. I want my consumer-grade GPU to be split up into the host machine and also into virtual machines, without worrying about resident neighbor cross-talk! Gosh that sounds like why I moved out of my apartment complex, actually.

The idea of having to assign a whole GPU via PCI passthrough is just asinine. I don't need to do that for my CPU, RAM, network, or storage. Why should I need to do it for my GPU?

Re: We were wrong about GPUs

#52

Post by yieldcrv »

Yes devs want LLMs, but also the price of inference compute plummeted 90% over the last 18 months, which is primarily in gpus

So it’s not just that openai and anthropic apis are good enough, they are also cheap enough, and still overpriced compared to the industry

Your GPU investment wont do as well as you thought, but also you are wasting time on security. If the end user and market doesnt care then you can consider not caring as well. Worst case you can pay for any settlement with …. more gpu credits.

Re: We were wrong about GPUs

#54

Post by unethical_ban »

reilly3000 wrote: post #41

I shelled out for a 4090 when they came out thinking it would be the key factor for running local llms. It turns out that anything worth running takes way more than 24GB VRAM. I would have been better off with 2+ 3090s and a custom power supply. It’s a pity because I thought it would be a great solution for coding and a home assistant, but performance and quality isn’t there yet for small models (afaik). Perhaps DIGI…

I haven't tested programming tasks with a local LLM vs. say, Claude 3.5. But it is nice to be able to run 14-32B LLMs locally and get an instant response. I have a single 3090.

Re: We were wrong about GPUs

#55

Post by 01HNNWZ0MV43FF »

reilly3000 wrote: post #41

I shelled out for a 4090 when they came out thinking it would be the key factor for running local llms. It turns out that anything worth running takes way more than 24GB VRAM. I would have been better off with 2+ 3090s and a custom power supply. It’s a pity because I thought it would be a great solution for coding and a home assistant, but performance and quality isn’t there yet for small models (afaik). Perhaps DIGI…

Gosh. Good thing I haven't bought a GPU in almost a decade. With a little luck I'll catch this wave on the back end. I haven't had to learn web or mobile development thoroughly either

Re: We were wrong about GPUs

#57

Post by mmastrac »

It's really a shame GPU slices aren't a thing -- a monthly cost of $1k for "a GPU" is just so far outside of what I could justify. I guess it's not terrible if I can batch-schedule a mega-gpu for an hour a day to catch up on tasks, but then I'm basically still looking at nearly $50/month.

I don't know exactly what type of cloud offering would satisfy my needs, but what's funny is that attaching an AMD consumer GPU to a Raspberry Pi is probably the most economical approach for a lot of problems.

Maybe something like a system where I could hotplug a full GPU into a system for a reservation of a few minutes at a time and then unplug it and let it go back into a pool?

FWIW it's that there's a large number of ML-based workflows that I'd like to plug into progscrape.com, but it's been very difficult to find a model that works without breaking the hobby-project bank.

Re: We were wrong about GPUs

#58

Post by sergiotapia »

you guys have all this juicy GPU and infrastructure. why not offer models as apis?

i would pay to have apis for:

sam2, florence, blip, flux 1.1, etc.

whatever use case I would have reached Fly for on GPU, i can't justify _not_ using Replicate. maybe Fly can do better offer premium queues for that with their juicy infra?

you're right! as a software dev I see dockerization and foisting these models as a burden, not a necessity.

Re: We were wrong about GPUs

#59

Post by latchkey »

There is no market for MIG in the cloud. People talk about it a lot, but in reality, nobody wants a partial GPU (at least not paying for it).

One interesting thing about all this is that 1 GPU / 1 VM doesn't work today with AMD GPUs like MI300x. You can't do pcie passthrough, but AMD is working on adding it to ROCm. We plan to be one of the first to offer this.

Re: We were wrong about GPUs

#60

Post by tptacek »

sergiotapia wrote: post #58

you guys have all this juicy GPU and infrastructure. why not offer models as apis? i would pay to have apis for: sam2, florence, blip, flux 1.1, etc. whatever use case I would have reached Fly for on GPU, i can't justify _not_ using Replicate. maybe Fly can do better offer premium queues for that with their juicy infra? you're right! as a software dev I see dockerization and foisting these models as a burden, not a n…

Someone should do that! Doesn't need to be us, though.
Post reply on HN