Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

431–440 of 604 posts

Re: We were wrong about GPUs

#431

Post by cullenking »

remram wrote: post #200
BobbyTables2 wrote: post #179

Earlier quoted context omitted.

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

Hey fellow k8s+ceph on bare metaler! We only have a 13 machine rack and 350tb of raw storage. No major issues with ceph after 16.x and all nvme storage though.

Re: We were wrong about GPUs

#432

Post by stevenfoster »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

This is one of the truest comments I have ever read on here

Re: We were wrong about GPUs

#433

Post by Kwpolska »

> Instead, we burned months trying (and ultimately failing) to get Nvidia’s host drivers working to map virtualized GPUs into Intel Cloud Hypervisor. At one point, we hex-edited the closed-source drivers to trick them into thinking our hypervisor was QEMU.

What do Nvidia’s lawyers think of this? There are some things that best not mentioned in a blog post, and this is one of them.

Re: We were wrong about GPUs

#434

Post by Rury »

raziel2p wrote: post #306
Rury wrote: post #219

Earlier quoted context omitted.

Yeah, I remember watching a video of (I think?) a European professor who helped with an issue devs were having in developing The Witness. Turns out they had a large algorithm they developed in high level code (~2000 lines of code? can't remember) to place flora in the game world, which took minutes to process, and it was hampering productivity. He looked at it all, and redid almost all of it in something like Frankly…

what's your point exactly? what do you hope to achieve by "bringing it up" (I assume in your workplace)? most programmers are not able to solve a problem like that in 20 lines of assembly or whatever, and no amount of education or awareness is going to change that. acting as if they can is just going to come across as arrogant.

The point is exactly as the above post mentioned:

> Half the things they are fixing, if not more, are created by the abstractions in the first place

Unlike the above post though, in my experience, it's less often devs (at least the best ones) who want to keep moving away from the silicon, but more often management. Everywhere I have worked, management wants to avoid control over the lower-level workings of things and outsource or abstract it away. They then proceed to wonder why we struggle with the issues that we have, despite people who deal with these things trying to explain it to them. They seem to automatically assume that higher level abstractions are inherently better, and will lead to productivity gains, simply because you don't have to deal with the underlying workings of things. But the example I gave, is reason for why that that isn't always necessarily the case. Fact is, sometimes problems are better and more easily solved in a lower-level abstraction.

But as I had said, in my experience, management often wants to go the opposite way and often disallows us control over these things. So, as an engineer who wants to solve the problems as much as management or customers want their problems solved, hope to achieve by "bringing it up" in cases which seem appropriate, a change which empowers us to actually solve such problems.

Don't get me wrong though, I'm not saying lower-level is always the way to go. It always depends on the circumstances.

Re: We were wrong about GPUs

#435

Post by sgarland »

notatoad wrote: post #244
sgarland wrote: post #193

Earlier quoted context omitted.

If you don’t have staff to do that, you probably aren’t at the scale when you need them, and you’re needlessly adding complexity. It’s always baffling to me why people think that ECS or god forbid EKS is somehow easier than a few Linux boxes.

a few linux boxes is great when you're a solo dev looking to save money and manage things yourself, but it's a poor place to start scaling up from. not just technologically, but from an HR perspective. Kubernetes is something you can hire for. A couple of linux boxes running all your server code in the most efficient way possible might save you operational costs, but it resigns you to being the one who has to maintai…

> Kubernetes is something you can hire for.

I massively distrust Ops-adjacent people's technical abilities if they don't know Linux. Multiple datapoints at multiple companies of varying scale has shown this to be true.

That said, you're correct, and I absolutely hate it. People want to do managed services for everything, and they stare at you like you're insane if you suggest running something yourself.

Re: We were wrong about GPUs

#436

Post by ec109685 »

scosman wrote: post #368

> But inference latency just doesn’t seem to matter yet, so the market doesn’t care. This is a very strange statement to make. They are acting like inference today happens with freshly spun up VMs and model access over remote networks (and their local switching could save the day). It’s actually hitting clusters of hot machines with the model of choice already loaded into VRAM. In real deployments, latency can be sma…

I think what they mean about latency not mattering is that latency to the LLM provider doesn’t matter. So why run it yourself when there are API’s you can hit that provide a better overall experience (and seems to be dropping in cost 90% year over year).

Re: We were wrong about GPUs

#437

Post by arijo »

laggyluke wrote: post #328
arijo wrote: post #302

Earlier quoted context omitted.

Great opportunity for someone ballsy to write a book about kubernetes internals for the general engineering population. Bonus points for writing a basic implementation from first principles capturing the essence of the problem kubernetes really was meant to solve. The 100 pages kubernetes book, Andriy Burkov style.

You might be interested in this: https://github.com/kelseyhightower/kubernetes-the-hard-way It probably won't answer the "why" (although any LLM can answer that nowadays), but it will definitely answer the "how".

I actually took the time to read the tutorial and found it helpful.

Thanks for taking the time to share the walk through.

Re: We were wrong about GPUs

#438

Post by zweifuss »

Kwpolska wrote: post #433

> Instead, we burned months trying (and ultimately failing) to get Nvidia’s host drivers working to map virtualized GPUs into Intel Cloud Hypervisor. At one point, we hex-edited the closed-source drivers to trick them into thinking our hypervisor was QEMU. What do Nvidia’s lawyers think of this? There are some things that best not mentioned in a blog post, and this is one of them.

I imagine you can do this without any reverse engineering of Nvidia's drivers, but §2.3 of the NVIDIA Driver License Agreement makes it hard (not impossible).

Re: We were wrong about GPUs

#439

Post by EarthIsHome »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> The other group (increasingly large) just wants to `git push` and be done with it, and they're willing to spend a lot of (usually their employer's) money to have that experience. They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using.

lol, even understanding git is hard for them. Increasingly, software engineers don't want to learn their craft.

Re: We were wrong about GPUs

#440

Post by chucky_z »

remram wrote: post #200
BobbyTables2 wrote: post #179

Earlier quoted context omitted.

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

I consider a '60+ node' kubernetes cluster is very small. Kubernetes at that scale is genuinely excellent! At 6000, 60000, and 600000 nodes it becomes very different and goes from 'Hey, this is pretty great' to 'What have I done?' The maintenance costs of running more than a hundred clusters is incredibly nontrivial especially as a lot of folks end up taking something open-source and thinking they can definitely do a lot better (you can.... there's a lot of "but"s there though).
Post reply on HN