Live data from Hacker News

We were wrong about GPUs

fly.io

511–520 of 604 posts

Re: We were wrong about GPUs

#511

Earlier quoted context omitted.

I'm a big fan of fundamental knowledge, but I disagree somewhat with your statement. The thing that startups care most about is a product market fit. And finding that fit requires a lot of iteration and throw-away code. Once the dust settles and you have an initial user base, you can start looking into optimizations.

> Once the dust settles and you have an initial user base, you can start looking into optimizations. But people never do. Instead they just scale up, get more funding, rinse and repeat. It isn't until the bill gets silly that anyone bothers to consider it, and they usually then discover that no one knows how to optimize things other than code (maybe – I've worked with many devs who have no idea how to profile their c…

Because that's the right approach for that situation. A core skill in engineering is understanding tradeoffs, and in that case you want speed.

Outside of eng, nobody cares if your company has the prettiest, leanest infrastructure in the world. They care about product.

Re: We were wrong about GPUs

#512
I feel like one of the mistakes being made again and again in the virtualization space is not realizing there's a difference between a competitor possibly running containers on the same machine with your proprietary data, and Dave over in Customer Relations running a container on your same machine.

If Dave does something malicious, we know where Dave lives, and we can threaten his livelihood. If your competitor does it you have to prove it, and they are protected from snooping at least as much as you are so how are you going to do that? I insist that the mutually assured destruction of coworkers substantially changes the equation.

In a Kubernetes world you should be able to saturate a machine with pods from the same organization even across teams by default, and if you're worried that the NY office is fucking with the SF office in order to win a competition, well then there should be some non-default flags that change that but maybe cost you a bit more due to underprovisioning.

You got a machine where one pod needs all of the GPUs and 8 cores, great. We'll load up some 8 core low memory pods onto there until the machine is full.

Re: We were wrong about GPUs

#513

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

I almost started laughing at the same comment. Kubernetes is the last place to know what your code is doing. A VM or bare metal is more practical for the persona that OP described. The git pushers might want the container on k8s

Re: We were wrong about GPUs

#514

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Its the first thing that came to the mind of the person who wrote the comment, which is positively terrifying

Re: We were wrong about GPUs

#515

Earlier quoted context omitted.

The most surprising thing about M1 was the energy efficiency and price/performance point they hit. It had been known for a couple of years that the phone SOCs were getting really good, just that being passively cooled inside a phone case only allows them 1-2 seconds of max bursts.

Apple's chips are dramatically faster than any other kind. If you are single thread perf constrained and have the money, running workloads on Apple silicon can actually make sense.

> Apple's chips are dramatically faster than any other kind.

Any idea why? Is it because of some patent they hold?

Re: We were wrong about GPUs

#516

> We were wrong about Javascript edge functions, and I think we were wrong about GPUs. Actually, you're still wrong about JavaScript edge functions. CF Workers slap.

They were wrong for us. Cloudflare is in a much different position than we were in 2019 trying to get people to write new Javascript. Clearly, for us, running people's existing applications natively was the better call. We're not dunking on Cloudflare's model.

Re: We were wrong about GPUs

#517
post #200

Earlier quoted context omitted.

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

[deleted]

Re: We were wrong about GPUs

#518

Earlier quoted context omitted.

I was genuinely surprised that k8s turned out to actually be pretty straightforward and very sensible after years of never having anything to do with it and just hearing about it on the net. Turns out opinions are just like after all. That being said, what people tend to build on top of that foundation is a somewhat different story.

I asked chatgpt the other day to explain to me Kubernetes. I still don't understand it. Can you share with me what clicked with you, or resources that helped you?

What clicked with me is having ChatGPT go line by line through all of the YAML files generated for a simple web app—WordPress on Kubernetes. Doing that, I realized that Kubernetes basically takes a set of instructions on how to run your app and then follows them.

So, take an app like WordPress that you want to make “highly available.” Let’s imagine it’s a very popular blog or a newspaper website that needs to serve millions of pages a day. What would you do without Kubernetes?

Without Kubernetes, you would get yourself a cluster of, let’s say, four servers—one database server, two worker servers running PHP and Apache to handle the WordPress code, and finally, a front-end load balancer/static content host running Nginx (or similar) to take incoming traffic and route it to one of the two worker PHP servers. You would set up all of your servers, network them, install all dependencies, load your database with data, and you’d be ready to rock.

If all of a sudden an article goes viral and you get 10x your usual traffic, you may need to quickly bring online a few more worker PHP nodes. If this happens regularly, you might keep two extra nodes in reserve and spin them up when traffic hits certain limits or your worker nodes’ load exceeds a given threshold. You may even write some custom code to do that automatically. I’ve done all that in the pre-Kubernetes days. It’s not bad, honestly, but Kubernetes just solves a lot of these problems for you in an automated way. Think of it as a framework for your hosting infrastructure.

On Kubernetes, you would take the same WordPress app and split it into the same four functional blocks. Each would become a container. It can be a Docker container or a Containerd container—as long as it’s compatible with the Open Container Initiative, it doesn’t really matter. A container is just a set of files defining a lightweight Linux virtual machine. It’s lightweight because it shares its kernel with the underlying host it eventually runs on, so only the code you are actually running really loads into memory on the host server.

You don’t really care about the kernel your PHP runs on, do you? That’s the idea behind containers—each process runs in its own Linux virtual machine, but it’s relatively efficient because only the code you are actually running is loaded, while the rest is shared with the host. I called these things virtual machines, but in practice they are just jailed and isolated processes running on the host kernel. No actual hardware emulation takes place, which makes it very light on resources.

Just like you don’t care about the kernel your PHP runs on, you don’t really care about much else related to the Linux installation that surrounds your PHP interpreter and your code, as long as it’s secure and it works. To that end, the developer community has created a large set of container templates or images that you can use. For instance, there is a container specifically for running Apache and PHP—it only has those two things loaded and nothing else. So all you have to do is grab that container template, add your code and a few setting changes if needed, and you’re off to the races.

You can make those config changes and tell Kubernetes where to copy and place your code files using YAML files. And that’s really it. If you read the YAML files carefully, line by line, you’ll realize that they are nothing more than a highly specialized way of communicating the same type of instructions you would write to a deployment engineer in an email when telling them how to deploy your code.

It’s basically a set of instructions to take a specific container image, load code into it, apply given settings, spool it up, monitor the load on the cluster, and if the load is too high, add more nodes to the cluster using the same steps. If the load is too low, spool down some nodes to save money.

So, in theory, Kubernetes was supposed to replace an expensive deployment engineer. In practice, it simply shifted the work to an expensive Kubernetes engineer instead. The benefit is automation and the ability to leverage community-standard Linux templates that are (supposedly) secure from the start. The downside is that you are now running several layers of abstraction—all because Unix/Linux in the past had a very unhealthy disdain for statically linked code. Kubernetes is the price we pay for those bad decisions of the 1980s. But isn’t that just how the world works in general? We’re all suffering the consequences of the utter tragedy of the 1980s—but that’s a story for another day.

Re: We were wrong about GPUs

#520
post #392

Earlier quoted context omitted.

> If you have a system that's actually big or complex enough to warrant using Kubernetes (...) I think there's a degree of confusion over your understanding of what Kubernetes is. Kubernetes is a platform to run containerized applications. Originally it started as a way to simplify the work of putting together clusters of COTS hardware, but since then its popularity drove it to become the platform instead of an abstr…

I'm not sure what your point is.

> I'm not sure what your point is.

My point is that the mere notion of "a system that's actually big or complex enough to warrant using Kubernetes" is completely absurd, and communicates a high degree of complete cluelessness over the whole topic.

Do you know what's a system big enough for Kubernetes? It's a single instance of a single container. That's it. Kubernetes is a container orchestration system. You tell it to run a container, and it runs it. That's it.

See how silly it all becomes once you realize these things?

Post reply on HN