Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

491–500 of 604 posts

Re: We were wrong about GPUs

#491

Post by antihero »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

The former have the mentality of being independent, at the cost of their ability to produce a result as quickly. The latter are happy to be dependent, because the result is more important than the means. Obviously this is a spectrum.

Re: We were wrong about GPUs

#492

Post by lenkite »

remram wrote: post #200
BobbyTables2 wrote: post #179

Earlier quoted context omitted.

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

"Kubernetes is etcd, apiserver, and controllers....Sure it's a lot of YAML but I don't understand the rep."

Sir, I upvoted you for your wonderful sense of humour.

Re: We were wrong about GPUs

#493

Post by jacobyoder »

mmastrac wrote: post #57

It's really a shame GPU slices aren't a thing -- a monthly cost of $1k for "a GPU" is just so far outside of what I could justify. I guess it's not terrible if I can batch-schedule a mega-gpu for an hour a day to catch up on tasks, but then I'm basically still looking at nearly $50/month. I don't know exactly what type of cloud offering would satisfy my needs, but what's funny is that attaching an AMD consumer GPU to…

vultr has fractional GPUs you can get as a VPS. I think I was paying about $55/month to test one out.

Re: We were wrong about GPUs

#494

Post by tptacek »

tempaccount420 wrote: post #480
tptacek wrote: post #452

Earlier quoted context omitted.

Sure. If it sounds like we're saying "cloud GPUs are not a product anybody wants", absolutely not. They're just not a knockout hit for us .

But why not add an option to rent them out without too many abstractions?

Because that's not what we're in business to do.

Re: We were wrong about GPUs

#495

Post by BoiledCabbage »

sgarland wrote: post #460
vaylian wrote: post #348

Earlier quoted context omitted.

I'm a big fan of fundamental knowledge, but I disagree somewhat with your statement. The thing that startups care most about is a product market fit. And finding that fit requires a lot of iteration and throw-away code. Once the dust settles and you have an initial user base, you can start looking into optimizations.

> Once the dust settles and you have an initial user base, you can start looking into optimizations. But people never do. Instead they just scale up, get more funding, rinse and repeat. It isn't until the bill gets silly that anyone bothers to consider it, and they usually then discover that no one knows how to optimize things other than code (maybe – I've worked with many devs who have no idea how to profile their c…

> But people never do. Instead they just scale up, get more funding, rinse and repeat. It isn't until the bill gets silly that anyone bothers to consider it,

Yes because usually the other option is focus on those things you advocate for up front and then they go out of business before they get a chance to have the problems you're arguing against.

Re: We were wrong about GPUs

#496

Post by remram »

ffsm8 wrote: post #453
remram wrote: post #412

Earlier quoted context omitted.

Never had a problem with the CSI plugin, I had problems with the Ceph cluster itself. No, I wouldn't call Ceph part of Kubernetes. You definitely can run Kubernetes without running Ceph or any storage system, and you already rely on a distributed storage system if you use the cloud whether you use Kubernetes or not. So I wouldn't count this as added complexity from Kubernetes.

I'm not sure I can agree with that interpretation. CSI is basically an interface that has to be implemented. If you discount issues like that, you can safely say that it's impossible to have any issues with CSI, because it's always going to be with one of it's implementation. That feels a little disingenuous, but maybe that's just me.

So if you run Kubernetes in the cloud, you consider the entire cloud provider's block storage implementation to be part of Kubernetes too?

For example you'd say AWS EBS is part of Kubernetes?

Re: We were wrong about GPUs

#497

Post by jval43 »

unification_fan wrote: post #114
hypothesis wrote: post #79

Earlier quoted context omitted.

> They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using. This is baffling. What’s value proposition here? At some point customer will be directly asking an AI agent to create an app for them and it will take care of coding/deployment for them..

Some people became software developers because they like learning and knowing what they're doing, and why and how it works. Some people became software developers because they wanted to make easy money back when the industry was still advertising bootcamps (in order to drive down the cost of developers). Some people simply drifted into this profession by inertia. And everything in-between. From my experience there ar…

This is my experience as well. I answer many such calls from devs as part of my work.

I can usually tell at the end of a call which group they belong to. I've been wrong a few times too.

As long as they don't waste my time I'm fine with everyone, some people just have other priorities in life.

One thing I'd say is in my experience there are many competent and capable people in every group, but non-competent ones are extremely rare in the first group.

Re: We were wrong about GPUs

#498

Post by pas »

chucky_z wrote: post #440
remram wrote: post #200

Earlier quoted context omitted.

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

I consider a '60+ node' kubernetes cluster is very small. Kubernetes at that scale is genuinely excellent! At 6000, 60000, and 600000 nodes it becomes very different and goes from 'Hey, this is pretty great' to 'What have I done?' The maintenance costs of running more than a hundred clusters is incredibly nontrivial especially as a lot of folks end up taking something open-source and thinking they can definitely do a…

At that point you probably need a cluster of k8s clusters, no?

For anyone unfamiliar with this the "official limits" are here, and as of 1.32 it's 5000 nodes, max 300k containers, etc.

https://kubernetes.io/docs/setup/best-practices/cluster-larg...

Re: We were wrong about GPUs

#499

Post by zsoltkacsandi »

fmbb wrote: post #442
chucky_z wrote: post #440

Earlier quoted context omitted.

I consider a '60+ node' kubernetes cluster is very small. Kubernetes at that scale is genuinely excellent! At 6000, 60000, and 600000 nodes it becomes very different and goes from 'Hey, this is pretty great' to 'What have I done?' The maintenance costs of running more than a hundred clusters is incredibly nontrivial especially as a lot of folks end up taking something open-source and thinking they can definitely do a…

OK but the alternative if you think Kubernetes is too much magic when you want to operate hundreds of clusters with tens of thousands of nodes is? Some bash and Ansible and EC2? That is usually what Kubernetes haters suggest one does to simplify.

Hashicorp Nomad, Docker Swarm, Apache Mesos, AWS ECS?

I love that the Kubernetes lovers tend to forget that Kubernetes is just one tool, and they believe that the only possible alternative to this coolness is that sweaty sysadmins writing bash scripts in a dark room.

Re: We were wrong about GPUs

#500

Post by hassleblad23 »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

You lost me at "Kubernetes".
Post reply on HN