Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

211–220 of 604 posts

Re: We were wrong about GPUs

#211

Post by lolinder »

remram wrote: post #200
BobbyTables2 wrote: post #179

Earlier quoted context omitted.

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

Because the root comment is mostly but not quite right: there are indeed a large subset of developers that aren't interested in thinking about infrastructure, but there are many subcategories of those people, and many of them aren't fly.io customers. A large number of people who are in that category aren't happy to let someone else handle their infra. They're not interested in infra in the sense that they don't believe it should be more complicated than "start process on Linux box and set up firewall and log rotation".

For some applications these people are absolutely right, but they've persuaded themselves that that means it's the best way to handle all use cases, which makes them see Kubernetes as way more complex than is necessary, rather than as a roll-your-own ECS for those who would otherwise truly need a cloud provider.

Re: We were wrong about GPUs

#212

Post by jimmaswell »

anacrolix wrote: post #186
nicoburns wrote: post #112

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Yeah no I wouldn't touch Kubernetes with a 10' pole. Way too much abstraction.

If my understanding is right, the gist seems to be that you create one or more docker containers that your application can run on, describe the parameters they require e.g. ram size/cuda capability/when you need more instances, and kubernetes provisions them out to the machines available to it based on those parameters. It's abstract but very tractibly so IMO, and it seems like a sensible enough way to achieve load balancing if you keep it simple. I plan to try it out on some machines of mine just for fun/research soon.

Re: We were wrong about GPUs

#213

Post by docandrew »

remram wrote: post #200
BobbyTables2 wrote: post #179

Earlier quoted context omitted.

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

“etcd, apiserver, and controllers.”

…and containerd and csi plugins and kubelet and cni plugins and kubectl and kube-proxy and ingresses and load balancers…

Re: We were wrong about GPUs

#214

Post by the__alchemist »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

Where does this dichotomy between Kubernetes, and superficial understanding come from? It is not consistent with my experience, and I don't have speculation its origin.

Re: We were wrong about GPUs

#215

Post by ksec »

lifeisstillgood wrote: post #83

I have a timeline that I am still trying to work through but it goes like this : 2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud - the excess parallel compute from each generation is just shared out i…

>. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud

Single Thread execution, I assume you mean IPC or may be more accurately as PPC ( Performance Per Clock ) has improved steadily if you accounted for ARM design and not just x86. That is why M1 was so surprising to everyone because most (all) thought Geekbench score on Phone doesn't translate to Desktop and somehow M1 went from nonsense to breakthrough.

Clockspeed also went from 2Ghz to 5Ghz and we are pushing 4Ghz on Mobile Phone already.

And Moores law, in terms of transistor density ends when Intel couldn't deliver 10nm on time, so 2016 / 2017 give or take. But that doesn't mean transistor density is not improving.

Re: We were wrong about GPUs

#216

Post by thundergolfer »

aqueueaqueue wrote: post #73

> developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. Is there not a market for the kind data science stuff where GPUs help but you are not using an LLM. Like statistical models on large amounts of data and so on. Maybe fly.io customer base isn't that sort of user. But I was pushing a previous company to get AWS GPUs because it would save us money vs CPU for the workload.

There is a market but it most likely requires a thick software layer to enter the competitive space. Modal Labs, Anyscale, and Outerbounds are examples of companies competing for "data science stuff" and have thick software layers over the VMs.

Re: We were wrong about GPUs

#217

Post by intelVISA »

transpute wrote: post #24
tptacek wrote: post #2

We wrote all sorts of stuff this week and this is what gets to the front page. :P

> We burned months trying (and ultimately failing) to get Nvidia’s host drivers working to map virtualized GPUs into Intel Cloud Hypervisor... We think there’s probably a market for users doing lightweight ML work getting tiny GPUs. This is what Nvidia MIG does, slicing a big GPU into arbitrarily small virtual GPUs. But for fully-virtualized workloads, it’s not baked; we can’t use it. Near as we can tell, MIG gives y…

It is quite possible, bit surprised a minor systems detour would burn months but alas. This inspires me to figure out how to sell it, sounds expensive...

Re: We were wrong about GPUs

#218

Post by post-it »

calvinmorrison wrote: post #210
post-it wrote: post #192

Earlier quoted context omitted.

Yeah, I think this really exemplifies the "everyone more specialized than me doesn't get the bigger picture, and everyone less specialized than me is wasting their time" trope. Developers who don't want to deal with the nitty gritty in one area are dealing with it in another area. Everyone has 24 hours in a day.

The difference between a good developer and a bad is understanding the stack. Not necessarily an expert but I spend a lot of time debugging for random issues and it could be dns or a file locking issue or a network or a api or parsing EDI whatever. Most recently I found a bug in software that had to do with how Windows runs 32 bit mode on 64 bit. I've never used windows professionally and I have only had unix machine…

It's great that you were able to debug that. It may have come at an opportunity cost of being able to solve some more specialized problem within your domain.

In my job I develop a React Native app. I also need to have a decent understanding of iOS and Android native code. If I run into a bug related to how iOS runs 32 bit vs 64 bit software? Not my problem, we'll open a ticket with Apple and block the ticket in our system.

Re: We were wrong about GPUs

#219

Post by Rury »

y1n0 wrote: post #174
serviceberry wrote: post #101

Earlier quoted context omitted.

> The truth is we are constantly moving further and further away from the silicon. Are we? We're constantly changing abstractions, but we don't keep adding them all that often. Operating systems and high-level programming languages emerged in the 1960s. Since then, the only fundamentally new layer of abstraction were virtual machines (JVM, browser JS, hardware virtualization, etc). There's still plenty of hardware-sp…

At the company I work for, I routinely mock the software devs for solving every problem by adding yet another layer of abstraction. The piles of abstractions these people levy is mind numbingly absurd. Half the things they are fixing, if not more, are created by the abstractions in the first place.

Yeah, I remember watching a video of (I think?) a European professor who helped with an issue devs were having in developing The Witness. Turns out they had a large algorithm they developed in high level code (~2000 lines of code? can't remember) to place flora in the game world, which took minutes to process, and it was hampering productivity. He looked at it all, and redid almost all of it in something like Frankly though, when I bring stuff like this up, it feels like I'm being mocked than the other way around - like we're the minority. And sadly, I'm not sure if anything can ultimately be done about it. People just don't know what they don't know. Some things you can't tell people despite trying to, they just won't get it.

Re: We were wrong about GPUs

#220

Post by thundergolfer »

tptacek wrote: post #67
thundergolfer wrote: post #65

Earlier quoted context omitted.

I think it's referencing only the IPv4 block, but it is a bit confusing. It doesn't make sense to be ref'ing the GPUs because their value is definitely not durable.

I don't know about "durable", but they're not written off. There is absolutely a market for all this hardware.

Yeah fair enough. I think it's just the subjectivity of "durable" at play here. The value of the GPUs may halve in a single year (e.g. H100s), but they'll never* drop to zero in a month. That's at least some kind of durability, because you can get a transaction done in a month.

* never say never

Post reply on HN