Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

191–200 of 604 posts

Re: We were wrong about GPUs

#191

Post by sgarland »

gregmac wrote: post #156
emilsedgh wrote: post #80

Earlier quoted context omitted.

You described two points in an spectrum in which: One end is PaaS like Heroku, where you just git push. The other end is bare metal hosting. Every option you mentioned (VPS, Manages K8S, Self Hosted K8S, etc) they all fall somewhere between these two ends of the spectrum. If, a developer falls into any of these "groups" or has a preference/position on any of these solutions, they are just called juniors. Where you en…

> Where you end up in this spectrum is a matter of cost benefit. Nothing else. And that calculation always changes. This is where I see things too. When you start out, all your value comes from working on your core problem. eg: You'd be crazy to start a CRM software business by building your own physical datacenter. It makes sense to use a PaaS that abstracts as much away as possible for you so you can focus on the a…

You’re correct that it would be absurd to build a DC, but you left out the next-best thing, and the one that is VERY financially attractive: colo’ing. I can rent 1U for around $50-75/month, or if I want HA-ish (same rack in the same DC isn’t exactly HA, but it solves for hardware failure anyway), 5U would probably run $200-250/month or so, and that lets you run two nodes with HAProxy or what-have-you, sharing a virtual IP, fronting three worker nodes running K8s, or a Proxmox cluster, or whatever. The hardware is also stupidly cheap, because you don’t need anything remotely close to new, so for about $200/node, you’ll have more cores and memory than you know what to do with.

The DC will handle physical service for you if something breaks, you just pay for parts and labor.

All of this requires knowledge, of course, but it’s hardly an impossible task. Go look at what the more serious folk in r/homelab (or r/datacenter) are up to; it’ll surprise you.

Re: We were wrong about GPUs

#192

Post by post-it »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Yeah, I think this really exemplifies the "everyone more specialized than me doesn't get the bigger picture, and everyone less specialized than me is wasting their time" trope. Developers who don't want to deal with the nitty gritty in one area are dealing with it in another area. Everyone has 24 hours in a day.

Re: We were wrong about GPUs

#193

Post by sgarland »

dirtbag__dad wrote: post #171
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> they're willing to spend a lot of (usually their employer's) money May just be my naïveté, but I thought that something like ECS or EKS is much cheaper than an in-house k8 engineer.

If you don’t have staff to do that, you probably aren’t at the scale when you need them, and you’re needlessly adding complexity.

It’s always baffling to me why people think that ECS or god forbid EKS is somehow easier than a few Linux boxes.

Re: We were wrong about GPUs

#194

Post by sanderjd »

PaulHoule wrote: post #173
sanderjd wrote: post #145

Earlier quoted context omitted.

Really? There are plenty of valid criticisms of kubernetes, but this doesn't strike me as one of them. It gives you tons of control over all of this. That's a big part of why it's so complex!

This mainframe system from the 1990s was so much simpler https://www.ibm.com/docs/en/cics-ts/6.x?topic=sysplex-parall... even if it wasn't as scalable as Kube. One the other hand, a cluster of 32 CMOS mainframe could handle any commercial computing job that people were doing in the 1990s.

It's not simple but it's not opaque.

Re: We were wrong about GPUs

#195

Post by morphle »

washadjeffmad wrote: post #118
zozbot234 wrote: post #63

Earlier quoted context omitted.

The best bang for the buck on VRAM is a maxed out Mac Studio.

No one who claims this ever posts a benchmark Prompt eval is slow, inference for large models at high context is slow, training is limited and slow. It's better than not having anything, but we got rid of our M1 Max 192GBs after about a year.

What will you pay me for the benchmarks, for the professional knowledge and analysis?

I can post benchmarks for these Mac machines and clusters of Studios and M4 Mac Minis (see my other HN posts last month, the largest Mac cluster I can benchmark for you has 4 TB of ultrafast unified memory and around 9216 M4 cores).

Re: We were wrong about GPUs

#196

Post by c0balt »

Spooky23 wrote: post #138
_dark_matter_ wrote: post #74

Earlier quoted context omitted.

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…

I think that’s the same answer someone would say about an IBM mainframe in 1990. And just as wrong. I’ll use my stupid hobby home server stuff as an example. I tossed the old VMware box years ago. You know what I use now? Little HP t6x0 thin clients. They are crappy little x86 SoCs with m2 slots, up to 32GB memory and they can be purchased used for $40. They aren’t fast, but perform better than the cheaper AWS and GC…

A hyperscaler (or a cloud providers in general) does not only sell you compute in terms of a compute node but rather in compute as a service. There are some value adds, like e.g., AWS cloud services, but on a pure compute level you pay for elasticity and reliability. A comparison between a cloud provider and your homelab also needs to account for connectivity, which likely is in strong favor (latency/ reliability) of a cloud provider or DC compared to a office or home.

Re: We were wrong about GPUs

#197

Post by cmrdporcupine »

brap wrote: post #137
lifeisstillgood wrote: post #83

I have a timeline that I am still trying to work through but it goes like this : 2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud - the excess parallel compute from each generation is just shared out i…

>someone can run their code 1 million times faster than a competitor I’d bet most code we use every day spends most time just waiting for things like disk and network. Not to mention it’s probably inherently sequential.

Hell most code we use every day spends a huge portion of its time waiting for memory.

Re: We were wrong about GPUs

#198

Post by Aeolun »

tptacek wrote: post #67
thundergolfer wrote: post #65

Earlier quoted context omitted.

I think it's referencing only the IPv4 block, but it is a bit confusing. It doesn't make sense to be ref'ing the GPUs because their value is definitely not durable.

I don't know about "durable", but they're not written off. There is absolutely a market for all this hardware.

There’s certainly more retained value in the physical stuff than in developer time.

Re: We were wrong about GPUs

#199

Post by SkyPuncher »

sgarland wrote: post #184
SkyPuncher wrote: post #94

Earlier quoted context omitted.

I enjoy the details, but I don’t get paid to tell my executives how we’re running things. I get paid to ship customer facing value. Particularly at startups, it’s almost always more cost effective to hit that “scale up” button from our hosting provider than do any sort of actual system engineering. Eventually, someone goes “hey we could save $$$$ by doing XYZ” so we send someone on a systems engineering journey for a…

Wild idea: maybe if more devs had good fundamental knowledge to begin with, the good systems engineering could be done along the way.

It all takes time, mental energy, etc.

Different environments require different tradeoffs. The vast majority of startups will die before their systems engineering becomes a problem.

Re: We were wrong about GPUs

#200

Post by remram »

BobbyTables2 wrote: post #179
nicoburns wrote: post #112

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph.

Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to get used to, but why is it always presented as this insurmountable complexity?

I ran into a VXLAN checksum offload kernel bug once, but otherwise this thing is just solid. Sure it's a lot of YAML but I don't understand the rep.

Post reply on HN