Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

221–230 of 604 posts

Re: We were wrong about GPUs

#221

Post by loopholelabs »

mmastrac wrote: post #57

It's really a shame GPU slices aren't a thing -- a monthly cost of $1k for "a GPU" is just so far outside of what I could justify. I guess it's not terrible if I can batch-schedule a mega-gpu for an hour a day to catch up on tasks, but then I'm basically still looking at nearly $50/month. I don't know exactly what type of cloud offering would satisfy my needs, but what's funny is that attaching an AMD consumer GPU to…

I think time-slicing for GPUs is likely the solution here.

If you could checkpoint a GPU quickly enough it would be possible to run multiple isolated workloads on the same GPUs without any issues.

Re: We were wrong about GPUs

#222

Post by docandrew »

dirtbag__dad wrote: post #171
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> they're willing to spend a lot of (usually their employer's) money May just be my naïveté, but I thought that something like ECS or EKS is much cheaper than an in-house k8 engineer.

Move to EKS and you still need a k8s engineer, but one who also knows AWS, and you also pay the AWS premium for the hosting, egress, etc. It might make sense for your use case but I definitely wouldn’t consider it a cost-saving measure.

Re: We were wrong about GPUs

#223

Post by npn »

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs.

No, I want GPU. BERT models are still useful.

The point is your service is too expensive that only one or two months of renting is enough to build a PC from scratch and place it somewhere in your workplace to run 24/7. For applications that need GPU power, usually downtime or latency does not really matter. And you always add an extra server to ensure.

Re: We were wrong about GPUs

#224

Post by calvinmorrison »

post-it wrote: post #218
calvinmorrison wrote: post #210

Earlier quoted context omitted.

The difference between a good developer and a bad is understanding the stack. Not necessarily an expert but I spend a lot of time debugging for random issues and it could be dns or a file locking issue or a network or a api or parsing EDI whatever. Most recently I found a bug in software that had to do with how Windows runs 32 bit mode on 64 bit. I've never used windows professionally and I have only had unix machine…

It's great that you were able to debug that. It may have come at an opportunity cost of being able to solve some more specialized problem within your domain. In my job I develop a React Native app. I also need to have a decent understanding of iOS and Android native code. If I run into a bug related to how iOS runs 32 bit vs 64 bit software? Not my problem, we'll open a ticket with Apple and block the ticket in our s…

I guess I never have enough leverage to order Apple to fix stuff. I'm like water and gravity. It's just a random example though and I agree you do give up a lot by being a generalist. However for most people we don't do really new or hard problems. Its a lot of spaghetti

Re: We were wrong about GPUs

#225

Post by throwaway2037 »

Moru wrote: post #93
varenc wrote: post #71

Earlier quoted context omitted.

Aren’t we just continually moving up layers of abstractions? Most of the increasingly small group doesn’t concern itself with voltages, manually setting jumpers, hand-rolling assembly for performance-critical code, cache line alignment, raw disk sector manipulation, etc. I agree it’s worthwhile to understand things more deeply but developers slowly moving up layers of abstractions seems like it’s been a long term tre…

We certainly need abstractions for the first layer of the hardware. An abstraction of the abstraction can be useful if the first abstraction is very bad or very crude. But we are now at an abstraction of an abstraction x 8 or so. It's starting to get a bit over the top.

I disgree with your sentiment. One thing that is constant in my experience as a computer programmer, there are always "old" computer programmers complaining that there are too many abstractions.

Re: We were wrong about GPUs

#226

Post by ryuuseijin »

raylad wrote: post #209
ryuuseijin wrote: post #158

My heart stopped for a moment when reading the title. I'm glad they haven't decided to axe GPUs, because fly GPU machines are FANTASTIC! Extremely fast to start on-demand, reliable and although a little bit pricy but not unreasonably so considering the alternatives. And the DX is amazing! it's just like any other fly machine, no new set of commands to learn. Deploy, logs, metrics, everything just works out of the box…

I just looked at their pricing and they don't list any GPUs at all that I could find.

Search for A100 on this page: https://fly.io/docs/about/pricing/

Re: We were wrong about GPUs

#227

Post by aaron695 »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

[dead]

Re: We were wrong about GPUs

#228

Post by digdugdirk »

morphle wrote: post #195
washadjeffmad wrote: post #118

Earlier quoted context omitted.

No one who claims this ever posts a benchmark Prompt eval is slow, inference for large models at high context is slow, training is limited and slow. It's better than not having anything, but we got rid of our M1 Max 192GBs after about a year.

What will you pay me for the benchmarks, for the professional knowledge and analysis? I can post benchmarks for these Mac machines and clusters of Studios and M4 Mac Minis (see my other HN posts last month, the largest Mac cluster I can benchmark for you has 4 TB of ultrafast unified memory and around 9216 M4 cores).

I mean, I can't pay you anything, but that sounds interesting as hell. Are there any interesting use cases to massive amounts of memory outside of training?

Re: We were wrong about GPUs

#229

Post by akoculu »

dnavani wrote: post #85
akoculu wrote: post #3

I spent a month setting up serverless endpoint for a custom model last year with Runpod. It was expensive and unreliable, in addition to long cold boot times. The product was unusable even as a prototype, to cover the costs, I'd have to raise money first. In a different product, I was given some Google Cloud credits, which unlocked me to put the product in front of customer. This one also needed GPU but not as expens…

Give https://modal.com a spin -- email me at deven [at] modal.com and happy to help you get set up

thanks, I'll reach out!

Re: We were wrong about GPUs

#230

Post by cratermoon »

jimmaswell wrote: post #212
anacrolix wrote: post #186

Earlier quoted context omitted.

Yeah no I wouldn't touch Kubernetes with a 10' pole. Way too much abstraction.

If my understanding is right, the gist seems to be that you create one or more docker containers that your application can run on, describe the parameters they require e.g. ram size/cuda capability/when you need more instances, and kubernetes provisions them out to the machines available to it based on those parameters. It's abstract but very tractibly so IMO, and it seems like a sensible enough way to achieve load b…

It's systemd but distributed across multiple nodes and with containers instead of applications. Instead of .service files telling the init process how to start and and monitor executables, you have charts telling the controller how to start and monitor containers.
Post reply on HN