Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

281–290 of 604 posts

Re: We were wrong about GPUs

#281

Post by gopher_space »

montecarl wrote: post #259
beebaween wrote: post #61

Earlier quoted context omitted.

This is what services like Vast.ai are for - super cheap GPUs you just use as long as you need etc etc.

Do you think that you can use those machines for confidential workflows for enterprise use? I'm currently struggling to balance running inference workloads on expensive AWS instances where I can trust that data remains private vs using more inexpensive platforms.

What's your workload and timeline? I'm wondering how much of that workload could be handled in-house.

Re: We were wrong about GPUs

#282

Post by ketzo »

sgarland wrote: post #184
SkyPuncher wrote: post #94

Earlier quoted context omitted.

I enjoy the details, but I don’t get paid to tell my executives how we’re running things. I get paid to ship customer facing value. Particularly at startups, it’s almost always more cost effective to hit that “scale up” button from our hosting provider than do any sort of actual system engineering. Eventually, someone goes “hey we could save $$$$ by doing XYZ” so we send someone on a systems engineering journey for a…

Wild idea: maybe if more devs had good fundamental knowledge to begin with, the good systems engineering could be done along the way.

And if more systems engineers had more design knowledge, navigating the AWS console wouldn’t be like walking on hot coals. But it’s still $X0 billion/year business!

We’re all different at good things, and it’s usually better to lean into your strengths than it is to paper over your weaknesses.

We can wish everyone were good at everything, or we can try to actually get things done.

Re: We were wrong about GPUs

#283

Post by almostgotcaught »

mandevil wrote: post #232
almostgotcaught wrote: post #203

Earlier quoted context omitted.

> At this point, optimizing compilers are excellent the only people that say this are people who don't work on compilers. ask anyone that actually does and they'll tell you most compiler are pretty mediocre (tend to miss a lot of optimization opportunities), some compilers are horrendous, and a few are good in a small domain (matmul).

It's more that the God of Moore's Law have given us so many transistors that we are essentially always I/O blocked, so it effectively doesn't matter how good our assembly is for all but the most specialized of applications. Good assembly, bad assembly, whatever, the point is that your thread is almost always going to be blocked waiting for I/O (disk, network, human input) rather than something that a fancy optimizati…

> It's more that the God of Moore's Law have given us so many transistors that we are essentially always I/O blocked

this is again just more brash confidence without experience. you're wrong. this is a post about GPUs and so i'll tell you that as a GPU compiler engineer i spend my entire day (work day) staring/thinking about asm in order to affect register pressure and ilp and load/store efficiency etc.

> rather than something that a fancy optimization of the loop

a fancy loop optimization (pipelinig) can fix some problems (load/store efficiency) but create other problems (register pressure). the fundamental fact is NFL theorem applies here fully: you cannot optimize for all programs uniformly.

https://en.wikipedia.org/wiki/No_free_lunch_theorem

Re: We were wrong about GPUs

#285

Post by mrcwinn »

loloquwowndueo wrote: post #39
mrcwinn wrote: post #37

Has service reliability improved at all? I tried Fly at two different points in time and I’ve never had a worse experience with a service.

You didn’t say at which points in time so it’s kind of hard to say yes but I will say “yes, reliability has improved”.

Okay, I'll try a different question. How's reliability these days, lolo?

Re: We were wrong about GPUs

#286

Post by fiberhood »

mdaniel wrote: post #241
simonw wrote: post #153

Earlier quoted context omitted.

192GB is available for the M2 Mac Studio.

I was curious "how bad is it?" and it seems $5500-ish https://www.ebay.com/sch/i.html?_nkw=192gb+studio&_sop=15

$6500 depending on VAT. But 10-12 times M4 Mac mini's with 100 Gbps networking gives you triple the cores and 160 GB with 2.5 times the memeory bandwith if the sharding of the NN layers is done right.

Re: We were wrong about GPUs

#287

Post by yazaddaruvala »

ehnto wrote: post #260
throwaway2037 wrote: post #225

Earlier quoted context omitted.

I disgree with your sentiment. One thing that is constant in my experience as a computer programmer, there are always "old" computer programmers complaining that there are too many abstractions.

You cannot see any way in which we run out of possible abstraction layers? I think in the past we have assumed that was natural language, but I think natural language is a pretty poor programming language. What people actually want when they say that, is for someone to read all the nuance out of their mind and codify it. I don't think we actually have been abstracting new layers over the past day 5-10 years anyway. M…

We only run out of abstraction once there is stagnation and time to really bake.

As long as some new thing is being invented in our industry, a new abstraction will be needed because the old one just can’t quite flex enough while being backwards compatible.

Re: We were wrong about GPUs

#288

Post by lakomen »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

[dead]

Re: We were wrong about GPUs

#289

Post by lakomen »

nicoburns wrote: post #250
igmor wrote: post #245

Earlier quoted context omitted.

These components are very different in complexity and scope. Let's be real: a seasoned developer is mostly familiar with load balancers and ingress controllers, so this will be mostly about naming and context. I agree though once you learn about k8s it becomes less mysterious but that also means the author hasn't pushed it to the limits. Outages in the control plane could be pretty nasty and it is easy to have them b…

A really simple setup for many smaller organisations wouldn't have a load balancer at all.

[dead]

Re: We were wrong about GPUs

#290

Post by montecarl »

gopher_space wrote: post #281
montecarl wrote: post #259

Earlier quoted context omitted.

Do you think that you can use those machines for confidential workflows for enterprise use? I'm currently struggling to balance running inference workloads on expensive AWS instances where I can trust that data remains private vs using more inexpensive platforms.

What's your workload and timeline? I'm wondering how much of that workload could be handled in-house.

Just currently exploring how custom AI workflows (e.g. text to sql, custom report generation using private data) can help given the current SOTA. Looking to develop tooling over the next 3-6 months. I'd like to see what we can come up with before dropping $50-100k on hardware.
Post reply on HN