We were wrong about GPUs • Hacker News Board

Re: We were wrong about GPUs

#421

Post by themgt » Sat, Feb 15, 2025, 3:32 PM UTC

apitman wrote: post #404

dzikimarian wrote: post #398

Earlier quoted context omitted.

One may think Kubernetes is complex (I agree), but I haven't seen alternative that simultaneously allows to: * Host hundreds or thousands of interacting containers across multiple teams in sane manner * Let's you manage and understand how is it done in the full extent. Of course there are tons of organizations that can (and should) easily resign from one of these, but if you need both, there isn't better choice right…

But how many orgs need that scale?

You can run single-node k3s on a VM with 512MB of RAM and deploy your app with a hundred lines of JSON, and it inherits a ton of useful features that are managed in one place and can grow with your app if/as needed. These discussions always go in circles between Haters and Advocates:

* H: "kubernetes [at planetary scale] is too complex"

* A: "you can run it on a toaster and it's simpler to reason about than systemd + pile of bash scripts"

* H: "what's the point of single node kubernetes? I'll just SSH in and paste my bash script and call it a day"

* A: "but how do you scale/maintain that?"

* H: "who needs that scale?"

Re: We were wrong about GPUs

#422

Post by morphle » Sat, Feb 15, 2025, 3:47 PM UTC

Aurornis wrote: post #397

morphle wrote: post #248

Earlier quoted context omitted.

> No one who claims this ever posts a benchmark I meant to explain why no one ever posts a benchmark, it's expensive as hell to do a professional benchmark against accepted standards. Its several days work, very expensive rental of several pieces of $10K hardware, etc. You don't often hand that over for free. With my benchmark results some companies can save millions if they take my advice. >any interesting use cases…

> it's expensive as hell to do a professional benchmark against accepted standards. Its several days work, very expensive rental of several pieces of $10K hardware, etc When people casually ask for benchmarks in comments, they’re not looking for in-depth comparisons across all of the alternatives. They just want to see “Running Model X with quantization Y I get Z tokens per second”. > That's why I know how to benchma…

>They just want to see “Running Model X with quantization Y I get Z tokens per second”.

Influencers on Youtube will give them that [1] but its meaningless. If a benchmark is not part of an in-depth comparison than it doesn't mean anything and can't inform you on what hardware will run this software best.

These shallow benchmarks influencers post on youtube and twitter are not just meaningless but also take days to browse through. And they are influencers, they are meant to influence you and are therefore not honest or reliable.

[1] https://www.youtube.com/watch?v=GBR6pHZ68Ho

>but I don’t understand how you’re concluding that a Mac is the “second best option” to your $30K machine

I conclude that if you can't afford to develop custom chips than in certain cases a cluster of M4 Mac Mini's will be the fastest cheapest option. Cerebras Wafers or NVDIA GPUs have always been too expensive compared to custom chips or Mac Mini clusters, independent of the specific software workload.

I also meant to say that a cluster of $599 Mac Minis will outperform a $6500 M2 Ultra Mac Studio with 192GB and be half the price for higher performance and DRAM but only if you utilize the M4 Mac Mini aggregated 100 Gbps networking.

Re: We were wrong about GPUs

#423

Post by chatmasta » Sat, Feb 15, 2025, 3:50 PM UTC

kbolino wrote: post #394

sgarland wrote: post #191

Earlier quoted context omitted.

You’re correct that it would be absurd to build a DC, but you left out the next-best thing, and the one that is VERY financially attractive: colo’ing. I can rent 1U for around $50-75/month, or if I want HA-ish (same rack in the same DC isn’t exactly HA, but it solves for hardware failure anyway), 5U would probably run $200-250/month or so, and that lets you run two nodes with HAProxy or what-have-you, sharing a virtu…

For $200/month, I can have 2 ALBs, 2 ECS services, 2 CloudWatch log groups, and 2 RDS instances on AWS (one each for dev and prod) and a GitHub Team account with enough included runner minutes to cover most deployments. A colo is going to be more hassle, and I'll have to monitor more things (like system upgrades and intrusion attempts). I'd also have to amortize parts and labor as part of the cost, which is going to…

For $200/month and all that auxiliary infrastructure, those two RDS instances will be running on the equivalent compute power of an iPhone 5s…

Re: We were wrong about GPUs

#424

Post by epr » Sat, Feb 15, 2025, 3:52 PM UTC

Izkata wrote: post #320

epr wrote: post #251

Earlier quoted context omitted.

No, they are not. I'm not sure who started this whole container is just a process thing, but it's not a good analogy. Quite a lot of things you spin up containers for have multiple processes (databases, web servers, etc). Containers are inherently difficult to sum up in a sentence. Perhaps the most reasonable comparison is to liken them to a "lightweight" vm, but the reasons people use them are so drastically differe…

> I'm not sure who started this whole container is just a process thing, but it's not a good analogy. Quite a lot of things you spin up containers for have multiple processes (databases, web servers, etc). It came from how Docker works, when you start a new container it runs a single process in the container, as defined in the Dockerfile. It's a simplification of what containers are capable of and how they do what th…

If a container is "a process", then an entire linux/unix os (pid 1) is simply "a process"

Re: We were wrong about GPUs

#425

Post by epr » Sat, Feb 15, 2025, 3:56 PM UTC

pclmulqdq wrote: post #292

epr wrote: post #251

Earlier quoted context omitted.

No, they are not. I'm not sure who started this whole container is just a process thing, but it's not a good analogy. Quite a lot of things you spin up containers for have multiple processes (databases, web servers, etc). Containers are inherently difficult to sum up in a sentence. Perhaps the most reasonable comparison is to liken them to a "lightweight" vm, but the reasons people use them are so drastically differe…

Having multiple processes under one user in an operating system is more akin to having multiple threads in one process than you think. The processes don't share a virtual memory space or kernel namespaces and they don't share PID namespaces, but that's pretty much all you get from process isolation (malware works because process isolation is relatively weak). The container adds a layer that goes around multiple proce…

> Having multiple processes under one user in an operating system is more akin to having multiple threads in one process than you think.

Not than I think. I'm well aware of how "tasks" work in Linux specifically, and am pretty comfortable working directly with clone.

Your explanation is great, but I intentionally went out of my way to not explain it and instead give a simple analogy. The entire point was that it's difficult to summarize.

Re: We were wrong about GPUs

#426

Post by epr » Sat, Feb 15, 2025, 4:00 PM UTC

chupasaurus wrote: post #407

epr wrote: post #251

Earlier quoted context omitted.

No, they are not. I'm not sure who started this whole container is just a process thing, but it's not a good analogy. Quite a lot of things you spin up containers for have multiple processes (databases, web servers, etc). Containers are inherently difficult to sum up in a sentence. Perhaps the most reasonable comparison is to liken them to a "lightweight" vm, but the reasons people use them are so drastically differe…

> Containers are inherently difficult to sum up in a sentence. Super easy if we talk about Linux. It's a process tree being spawned inside it's own set of kernel namespaces, security measures and a cgroup to provide isolation from the rest of the system.

If someone doesn't understand "container", I'm supposed to expect them to understand all the namespaces and their uses, cgroups, and the nitty gritty of the wimpy security isolation? You are proving my point that it's tough to summarize by using a bunch more terms that are difficult to summarize.

Once you recursively expand all the concepts, you will have multiple dense paragraphs, which don't "summarize" anything, but instead provide full explanations.

Re: We were wrong about GPUs

#427

Post by arijo » Sat, Feb 15, 2025, 4:00 PM UTC

adhamsalama wrote: post #409

arijo wrote: post #302

Earlier quoted context omitted.

Great opportunity for someone ballsy to write a book about kubernetes internals for the general engineering population. Bonus points for writing a basic implementation from first principles capturing the essence of the problem kubernetes really was meant to solve. The 100 pages kubernetes book, Andriy Burkov style.

Kubernetes in Action book is very good.

I actually have the book and I agree it is very good.

Re: We were wrong about GPUs

#428

Post by tarsinge » Sat, Feb 15, 2025, 4:06 PM UTC

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

It’s not about wanting, it’s about what the job asks for. As a self employed engineer I am paid to solve business problems in an efficient way. Most of the time it just make more business sense for the client and for me to pay to just have to git push if there is no performance challenges needing custom infrastructure.

Re: We were wrong about GPUs

#429

Post by immibis » Sat, Feb 15, 2025, 4:09 PM UTC

yazaddaruvala wrote: post #287

ehnto wrote: post #260

Earlier quoted context omitted.

You cannot see any way in which we run out of possible abstraction layers? I think in the past we have assumed that was natural language, but I think natural language is a pretty poor programming language. What people actually want when they say that, is for someone to read all the nuance out of their mind and codify it. I don't think we actually have been abstracting new layers over the past day 5-10 years anyway. M…

We only run out of abstraction once there is stagnation and time to really bake. As long as some new thing is being invented in our industry, a new abstraction will be needed because the old one just can’t quite flex enough while being backwards compatible.

One problem is that we're building abstractions on top of older abstractions when that isn't most efficient. Suppose we write an HTTP server that can call CGI programs. Then we implement PHP as a CGI program. Then we write a Lua interpreter in PHP. Then we write our website in Lua.

Some of those levels are useful. Some of them are redundant. We should embed a Lua interpreter in our webserver and delete two levels of abstraction.

(I'm not aware of any actual Lua interpreter written in PHP, but it's representative of the kinds of stacks that do exist out there)

Re: We were wrong about GPUs

#430

Post by sgarland » Sat, Feb 15, 2025, 4:10 PM UTC

mdaniel wrote: post #238

sgarland wrote: post #193

Earlier quoted context omitted.

If you don’t have staff to do that, you probably aren’t at the scale when you need them, and you’re needlessly adding complexity. It’s always baffling to me why people think that ECS or god forbid EKS is somehow easier than a few Linux boxes.

Because the force multiplier of a good DX way outweighs the occasional nonsense from having to do k8s upgrades or troubleshooting For example: how do you roll out a new release of your product? In sane setups, it's often $(helm upgrade --install ...), which is itself often run either in-cluster by watching a git managed descriptor, or in CI on merge to a release branch/tag How does your developer get logs? Maybe it's…

> For example: how do you roll out a new release of your product?

It's pretty easy to accomplish that with docker compose if you have containers, but you can also use systemd and some bash scripts to accomplish the same thing. Admittedly this would only affect a single node, but it's also possible to manage multiple nodes without using K8s / Nomad.

> How does your developer get logs?

fluentd

> How do you jump into the execution environment for your workload, to do more advanced debugging?

ssh

> how do you audit what was done, to prevent config drift

Assuming you're pulling down releases from a git repo, git diff can be used to detect changes, and you can then opt to either generate a patch file and send it somewhere, or just reset to HEAD. For server settings, any config management tool, e.g. puppet.

> how do you authenticate the right developer at the right time with access to the right machine without putting root's public key file in a spreadsheet somewhere

freeipa

I'm not saying any of this is better than K8s. I'm saying that, IMO, the above can be simpler to reason about for small setups, and has a lot less resource overhead. Now, if you're already comfortable administering and troubleshooting K8s (which is quite a bit different than using it), and you have no background in any of the above, then sure, K8s is probably easier. But if you don't know this stuff, there's a good chance you don't have a solid background in Linux administration, which means when your app behaves in strange ways (i.e. not an application bug per se, but how it's interacting with Linux) or K8s breaks, you're going to struggle to figure out why.