Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

131–140 of 604 posts

Re: We were wrong about GPUs

#131

Post by pdntspa »

_dark_matter_ wrote: post #74
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…

> why do the details matter?

This statement encapsulates nearly everything that I think is wrong with software development today. Captured by MBA types trying to make a workforce that is as cheap and replaceable as possible. Details are simply friction in a machine that is obsessed with efficiency to the point of self-immolation. And yet that is the direction we are moving in.

Details matter, process matters, experience and veterancy matters. Now more than ever.

Re: We were wrong about GPUs

#132

Post by andrewstuart »

mmastrac wrote: post #57

It's really a shame GPU slices aren't a thing -- a monthly cost of $1k for "a GPU" is just so far outside of what I could justify. I guess it's not terrible if I can batch-schedule a mega-gpu for an hour a day to catch up on tasks, but then I'm basically still looking at nearly $50/month. I don't know exactly what type of cloud offering would satisfy my needs, but what's funny is that attaching an AMD consumer GPU to…

Nvidia does not want slicing.

Re: We were wrong about GPUs

#133

Post by ignoramous »

jwr wrote: post #81
reilly3000 wrote: post #41

I shelled out for a 4090 when they came out thinking it would be the key factor for running local llms. It turns out that anything worth running takes way more than 24GB VRAM. I would have been better off with 2+ 3090s and a custom power supply. It’s a pity because I thought it would be a great solution for coding and a home assistant, but performance and quality isn’t there yet for small models (afaik). Perhaps DIGI…

I started buying Macs with more memory, no regrets. An M4 Max with 64GB (in a laptop, no less!) runs most small models comfortably (but get 96GB or more if you really intend to use 70B models regularly). And when I'm not running LLMs, the memory is useful for other stuff.

> And when I'm not running LLMs, the memory is useful for other stuff

Let's be honest, the other stuff is just Chrome: Tell me 96gb is enough?

Re: We were wrong about GPUs

#134

Post by emilsedgh »

tptacek wrote: post #106
emilsedgh wrote: post #99

Earlier quoted context omitted.

Ok, let me rephrase this: > Their audience has always been, and will always be application developers who prefer to do nothing low-level. How did they forget this? to this: Their audience has always been, and will always be application developers who prefer to do nothing except to build their main product. > Our primary DX is a CLI. One of our defining features is hardware isolation. To use us, you have to manage Doc…

No. It is definitely not the case that the modal developer today needs to know Docker. If only! It's a huge pain point for us with the PaaS customer cohort.

So why are you guys not becoming Heroku compatible?

https://github.com/gliderlabs/herokuish

Something like this + Procfile support should allow you to gobble up Heroku customers [like us] quickly since they've been stagnating for long, no?

Re: We were wrong about GPUs

#135

Post by llm_trw »

_dark_matter_ wrote: post #74
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…

> If the cost of an easily managed solution is low enough

Because vertical scaling is now large enough that I can run all of twitter/amazon on one single large server. And if I'm wrong now, in a decade I won't be.

Compute power grows exponentially, but business requirements do not.

Re: We were wrong about GPUs

#136

Post by nitwit005 »

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs.

My current company has some finance products. There was machine learning used for things like fraud and risk before the recent AI excitement.

Our executives are extremely enthused with AI, and seemingly utterly uncaring that we were already using it. From what I can tell, they genuinely just want to see ChatGPT everywhere.

The fraud team announce they have a "new" AI based solution. I assume they just added a call to OpenAI somewhere.

Re: We were wrong about GPUs

#137

Post by brap »

lifeisstillgood wrote: post #83

I have a timeline that I am still trying to work through but it goes like this : 2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud - the excess parallel compute from each generation is just shared out i…

>someone can run their code 1 million times faster than a competitor

I’d bet most code we use every day spends most time just waiting for things like disk and network. Not to mention it’s probably inherently sequential.

Re: We were wrong about GPUs

#138

Post by Spooky23 »

_dark_matter_ wrote: post #74
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…

I think that’s the same answer someone would say about an IBM mainframe in 1990. And just as wrong.

I’ll use my stupid hobby home server stuff as an example. I tossed the old VMware box years ago. You know what I use now? Little HP t6x0 thin clients. They are crappy little x86 SoCs with m2 slots, up to 32GB memory and they can be purchased used for $40. They aren’t fast, but perform better than the cheaper AWS and GCP instances.

In that a trivial use case? Absolutely. Now move from $30 to about $2000. Buy a Mac Mini. It’s a powerful arm soc with ridiculously fast storage and performance. Probably more compute than a small/mid size company computer room a few years ago and more performant than a $1M SAN a decade ago.

6G will bring 10gig cellular.

Hyperscalers datacenters are the mainframe of 2025.

Re: We were wrong about GPUs

#139

Post by brightball »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

I am the former. I also make cost benefit based decisions that involve time. Unless I have very specific configuration needs, the git push option lets me focus on what my users care about and gives me one less thing that I need to spend my time on.

Increasingly, Fly even lets you dip into most complex configurations too.

I’ve got no issue with using Tofu and Ansible to manage my own infrastructure but it takes time to get it right and it’s typically not worth the investment early on in the lifecycle.

Re: We were wrong about GPUs

#140

Post by bolognafairy »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Lol. Yes. I scoffed.
Post reply on HN