Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

441–450 of 604 posts

Re: We were wrong about GPUs

#441

Post by kbolino »

chatmasta wrote: post #423
kbolino wrote: post #394

Earlier quoted context omitted.

For $200/month, I can have 2 ALBs, 2 ECS services, 2 CloudWatch log groups, and 2 RDS instances on AWS (one each for dev and prod) and a GitHub Team account with enough included runner minutes to cover most deployments. A colo is going to be more hassle, and I'll have to monitor more things (like system upgrades and intrusion attempts). I'd also have to amortize parts and labor as part of the cost, which is going to…

For $200/month and all that auxiliary infrastructure, those two RDS instances will be running on the equivalent compute power of an iPhone 5s…

Pretty much. I don't really see the problem though. You're also getting regular snapshots, which is yet another thing you have to build in that colo setup (and where are your backups going?). This is not for personal projects or shoestring-budget non-profits where you're willing to volunteer time, it's for businesses and decent paying work, where two hundred dollars is one man-hour of totally compensated labor.

Re: We were wrong about GPUs

#442

Post by fmbb »

chucky_z wrote: post #440
remram wrote: post #200

Earlier quoted context omitted.

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to ge…

I consider a '60+ node' kubernetes cluster is very small. Kubernetes at that scale is genuinely excellent! At 6000, 60000, and 600000 nodes it becomes very different and goes from 'Hey, this is pretty great' to 'What have I done?' The maintenance costs of running more than a hundred clusters is incredibly nontrivial especially as a lot of folks end up taking something open-source and thinking they can definitely do a…

OK but the alternative if you think Kubernetes is too much magic when you want to operate hundreds of clusters with tens of thousands of nodes is?

Some bash and Ansible and EC2? That is usually what Kubernetes haters suggest one does to simplify.

Re: We were wrong about GPUs

#443

Post by formerly_proven »

remram wrote: post #414
cuu508 wrote: post #299

Earlier quoted context omitted.

Hence > Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

CSI and CNI do about as much magic as `docker volume` and `docker network`. People act like their web framework and SQL connection pooler and stuff are so simple, while Kubernetes is complex and totally inscrutable for mortals, and I don't get it. It has a couple of moving parts, but it is probably simpler overall than SystemD.

I was genuinely surprised that k8s turned out to actually be pretty straightforward and very sensible after years of never having anything to do with it and just hearing about it on the net. Turns out opinions are just like after all.

That being said, what people tend to build on top of that foundation is a somewhat different story.

Re: We were wrong about GPUs

#444

Post by freedomben »

post-it wrote: post #218
calvinmorrison wrote: post #210

Earlier quoted context omitted.

The difference between a good developer and a bad is understanding the stack. Not necessarily an expert but I spend a lot of time debugging for random issues and it could be dns or a file locking issue or a network or a api or parsing EDI whatever. Most recently I found a bug in software that had to do with how Windows runs 32 bit mode on 64 bit. I've never used windows professionally and I have only had unix machine…

It's great that you were able to debug that. It may have come at an opportunity cost of being able to solve some more specialized problem within your domain. In my job I develop a React Native app. I also need to have a decent understanding of iOS and Android native code. If I run into a bug related to how iOS runs 32 bit vs 64 bit software? Not my problem, we'll open a ticket with Apple and block the ticket in our s…

I don't disagree with you, but I do think it's important to acknowledge that this approach requires someone else to do it. If you're at a big company where there are tons of specialists, then perhaps this is just fine because there is someone available to do it for you. If you find yourself in a different situation, however, where you don't have that other specialist, you could end up significantly blocked for a period of time. If whatever you're working on is not important and can afford to be blocked, then again no problem, but I've been in many situations where what I was doing absolutely had to work and had to work on a timetable. If I had to offload the work to someone else because I wasn't capable, it would have meant disaster.

Re: We were wrong about GPUs

#445

Post by KronisLV »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Maybe not Kubernetes, but what about Docker Compose or Docker Swarm? Having each app be separate from the rest of the server, with easily controllable storage, networking, resource limits, restarts, healthchecks, configuration and other things. It's honestly a step up from well crafted cgroups and systemd services etc. (also because it comes in a coherent package and a unified description of environments) while the caveats and shortcomings usually aren't great enough to be dealbreakers.

But yeah, the argument could have as well just said running code on a VPS directly, because that also gives you a good deal of control.

Re: We were wrong about GPUs

#446

Post by dilyevsky »

nicoburns wrote: post #250
igmor wrote: post #245

Earlier quoted context omitted.

These components are very different in complexity and scope. Let's be real: a seasoned developer is mostly familiar with load balancers and ingress controllers, so this will be mostly about naming and context. I agree though once you learn about k8s it becomes less mysterious but that also means the author hasn't pushed it to the limits. Outages in the control plane could be pretty nasty and it is easy to have them b…

A really simple setup for many smaller organisations wouldn't have a load balancer at all.

No code at all even - just use excel

Re: We were wrong about GPUs

#447

Post by efitz »

apitman wrote: post #404
dzikimarian wrote: post #398

Earlier quoted context omitted.

One may think Kubernetes is complex (I agree), but I haven't seen alternative that simultaneously allows to: * Host hundreds or thousands of interacting containers across multiple teams in sane manner * Let's you manage and understand how is it done in the full extent. Of course there are tons of organizations that can (and should) easily resign from one of these, but if you need both, there isn't better choice right…

But how many orgs need that scale?

Most of the ones that are profitable for cloud providers.

Re: We were wrong about GPUs

#448

Post by tptacek »

pk-protect-ai wrote: post #334
nicoburns wrote: post #112

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Fly.io probably runs it on Kubernetes as well. It can be something in the middle, like RunPod. If you select 8 GPUs, you'll get a complete host for yourself. Though there is a lot of stuff lacking at RunPod too. But Fly.io... First of all, I've never heard about this one. Second, the variety of GPUs is lacking. There are only 3 types, and the L40S on Fly.io is 61.4% more expensive than on RunPod. So I would say it is…

We do not use K8s.

Re: We were wrong about GPUs

#449

Post by vagrantJin »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> I'm increasingly coming to the view that there is a big split among "software developers" and AI is exacerbating it

This is admittedly low effort but the vast majority of devs are paid wages to "write CRUD, git push and magic" their way to the end of the month. The company does not afford them the time and privilege of sitting down and analyzing the code with a fine comb. An abstraction that works is good enough.

The seasoned seniors get paid much more and afforded leeway to care about what is happening in the stack, since they are largely responsible for keeping things running. I'm just pointing out it might merely be a function of economics.

Re: We were wrong about GPUs

#450

Post by alexjplant »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Based on the following I think they also meant _how_ the code is running:

> The other group (increasingly large) just wants to `git push` and be done with it, and they're willing to spend a lot of (usually their employer's) money to have that experience. They don't want to have to understand DNS, linux, or anything else beyond whatever framework they are using.

I'm a "full full-stack" developer because I understand what happens when you type an address into the address bar and hit Enter - the DNS request that returns a CNAME record to object storage, how it returns an SPA, the subsequent XHR requests laden with and cookies and other goodies, the three reverse proxies they have to flow through to get to before they get to one of several containers running on a fleet of VMs, the environment variable being injected by the k8s control plane from a Secret that tells the app where the Postgres instance is, the security groups that allow tcp/5432 from the node server to that instance, et cetera ad infinitum. I'm not hooking debuggers up to V8 to examine optimizations or tweaking container runtimes but I can speak intelligently to and debug every major part of a modern web app stack because I feel strongly that it's my job to be able to do so (and because I've worked places where if I didn't develop that knowledge then nobody would have).

I can attest that this type of thinking is becoming increasingly rare as our industry continues to specialize. These considerations are now often handled by "DevOps Engineers" who crank out infra and seldom write code outside of Python and bash glue scripts (which is the antithesis to what DevOps is supposed to be, but I digress). I find this unfortunate because this results in teams throwing stuff over the wall to each other which only compounds the hand-wringing when things go wrong. Perhaps this is some weird psychopathology of mine but I sleep much better at night knowing that if I'm on the hook for something I can fix it once it's out in the wild, not just when I'm writing features and debugging it locally.

Post reply on HN