Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

151–160 of 604 posts

Re: We were wrong about GPUs

#151

Post by cyberax »

nserrino wrote: post #142
cyberax wrote: post #40

Hah. We're doing AI, but we're doing vision-based stuff and not LLMs. For us, the problem has been deploying models. Google and AWS helpfully offered their managed LLM AI services, but they don't really have anything terribly more useful than just machines with GPUs. Which are expensive. I'm going to check fly.io...

What kind of models are you deploying and what type of problems are you having with deploying them?

Aerial imagery analysis. It's a mix of classic computer vision and AI for some purposes.

Re: We were wrong about GPUs

#152

Post by simonw »

ignoramous wrote: post #133
jwr wrote: post #81

Earlier quoted context omitted.

I started buying Macs with more memory, no regrets. An M4 Max with 64GB (in a laptop, no less!) runs most small models comfortably (but get 96GB or more if you really intend to use 70B models regularly). And when I'm not running LLMs, the memory is useful for other stuff.

> And when I'm not running LLMs, the memory is useful for other stuff Let's be honest, the other stuff is just Chrome: Tell me 96gb is enough?

Firefox and VS Code for me: I have 64GB and I can't run Llama 70B locally without closing a ton of windows and tabs first!

Re: We were wrong about GPUs

#153

Post by simonw »

sepositus wrote: post #108
fabiensanglard wrote: post #69

Earlier quoted context omitted.

Absolutely! I have been playing with Ollama on a Macbook Pro 192 GiB RAM and it is able to run most models whereas my 3090 runs our of RAM.

Do you mean 128GB? Not aware of any variant of the Macbook Pro with that much RAM.

192GB is available for the M2 Mac Studio.

Re: We were wrong about GPUs

#154

Post by danstewart_ »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

I don’t think this comment does justice to fly.io.

They have incredible defaults that can make it as simple as just running ‘git push’ but there isn’t really any magic happening, it’s all documented and configurable.

Re: We were wrong about GPUs

#156

Post by gregmac »

emilsedgh wrote: post #80
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

You described two points in an spectrum in which: One end is PaaS like Heroku, where you just git push. The other end is bare metal hosting. Every option you mentioned (VPS, Manages K8S, Self Hosted K8S, etc) they all fall somewhere between these two ends of the spectrum. If, a developer falls into any of these "groups" or has a preference/position on any of these solutions, they are just called juniors. Where you en…

> Where you end up in this spectrum is a matter of cost benefit. Nothing else. And that calculation always changes.

This is where I see things too. When you start out, all your value comes from working on your core problem.

eg: You'd be crazy to start a CRM software business by building your own physical datacenter. It makes sense to use a PaaS that abstracts as much away as possible for you so you can focus on the actual thing that generates value.

As you grow, the high abstraction PaaS gets increasingly expensive, and at some point bubbles up to where it's the most valuable thing to work on. This typically means moving down a layer or two. Then you go back to improving your actual software.

You go through this a bunch of times, and over time grow teams dedicated to this work. Given enough time and continuous growth, it should eventually make sense to run your own data centers, or even build your own silicon, but of course very few companies get to that level. Instead most settle somewhere in the vast spectrum of the middle, with a mix of different services/components all done at different levels of abstraction.

Re: We were wrong about GPUs

#157

Post by wmf »

UncleOxidant wrote: post #46
reilly3000 wrote: post #41

I shelled out for a 4090 when they came out thinking it would be the key factor for running local llms. It turns out that anything worth running takes way more than 24GB VRAM. I would have been better off with 2+ 3090s and a custom power supply. It’s a pity because I thought it would be a great solution for coding and a home assistant, but performance and quality isn’t there yet for small models (afaik). Perhaps DIGI…

There was a post on r/localLlama the other day about a presentation by the company building Digits hardware for Nvidia. The gist was that Digits is going to be aimed at academic AI research folks and as such don't expect them to be available in large numbers (at least not for this first version). It was disappointing. Now I'm awaiting the AMD Strix Halo based systems.

Laptops with the same chip however...

Re: We were wrong about GPUs

#158

Post by ryuuseijin »

My heart stopped for a moment when reading the title. I'm glad they haven't decided to axe GPUs, because fly GPU machines are FANTASTIC!

Extremely fast to start on-demand, reliable and although a little bit pricy but not unreasonably so considering the alternatives.

And the DX is amazing! it's just like any other fly machine, no new set of commands to learn. Deploy, logs, metrics, everything just works out of the box.

Regarding the price: we've tried a well known cheaper alternative and every once in a while on restart inference performance was reduced by 90%. We never figured out why, but we never had any such problems on fly.

If I'm using a cheaper "Marketplace" to run our AI workloads, I'm also not really clear on who has access to our customer's data. No such issues with fly GPUs.

All that to say, fly GPUs are a game changer for us. I could wish only for lower prices and more regions, otherwise the product is already perfect.

Re: We were wrong about GPUs

#159

Post by wmf »

andrewstuart wrote: post #132
mmastrac wrote: post #57

It's really a shame GPU slices aren't a thing -- a monthly cost of $1k for "a GPU" is just so far outside of what I could justify. I guess it's not terrible if I can batch-schedule a mega-gpu for an hour a day to catch up on tasks, but then I'm basically still looking at nearly $50/month. I don't know exactly what type of cloud offering would satisfy my needs, but what's funny is that attaching an AMD consumer GPU to…

Nvidia does not want slicing.

Eh, they have MIG.

Re: We were wrong about GPUs

#160

Post by ignoramous »

mrkurt wrote: post #44
ksec wrote: post #14

Earlier quoted context omitted.

In the current days of AI I think spelling and grammar mistakes is perhaps a great way to tell it is still written by human...... ( Until AI copy this )

I do, in fact, instruct LLMs to make spelling and grammar mistakes when I have them reply to cold emails.

Do you run those LLMs on Fly? ;)
Post reply on HN