Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

501–510 of 604 posts

Re: We were wrong about GPUs

#501

Post by chromanoid »

sgarland wrote: post #463
chromanoid wrote: post #415

Earlier quoted context omitted.

Premature optimization may hit them hard. Overengineering is imo usually the bigger technical debt and a huge upfront cost as well. Well-thought out plans tend to become a sunken cost fallacy. Making room for changes is hard enough in XP like ways of working. When you have to tell your manager that half a year of careful plans and engineering can be thrown away, because of the new requirements, which emerge from late…

Infra should not require much in the way of redoing if it's done correctly. Foundational software's configuration like RDBMS schema, maybe, but I wouldn't classify that as infra per se. Seriously, I'm struggling to figure out how "we have servers that run containers / applications" would need to be redone just because the application changed.

Some things that can happen: Product gets canned. Customers want on premise in their data center. Usage spikes are too extreme and serverless is simply the cheapest option.

I would always recommend "serverless" monolith first with the option to develop with mocks locally/offline. That's imo the best risk/effort ratio.

Re: We were wrong about GPUs

#502

Post by dylan604 »

BobbyTables2 wrote: post #179
nicoburns wrote: post #112

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes… Just an “idle” Kubernetes system is a behemoth to comprehend…

> I have to wonder how many people actually understand Kubernetes.

I have to wonder how many people actually understand when to use K8s or docker. Docker is not a magic bullet, and can actually be a foot gun when it's not the right solution.

Re: We were wrong about GPUs

#503

Post by ffsm8 »

remram wrote: post #496
ffsm8 wrote: post #453

Earlier quoted context omitted.

I'm not sure I can agree with that interpretation. CSI is basically an interface that has to be implemented. If you discount issues like that, you can safely say that it's impossible to have any issues with CSI, because it's always going to be with one of it's implementation. That feels a little disingenuous, but maybe that's just me.

So if you run Kubernetes in the cloud, you consider the entire cloud provider's block storage implementation to be part of Kubernetes too? For example you'd say AWS EBS is part of Kubernetes?

In the context of this discussion, which is about the complexity of the k8s stack: yes.

Youre ultimately gonna have to use a storage of some form unless you're just a stateless service/keep the services with state out of k8s. That's why I'd include it, and the fact that you can use multiple storage backends, each with their own challenges and pitfalls makes k8s indeed quiet complex.

You could argue that multinode PaaS is always going to be complex, and frankly- I'd agree with that. But that was kinda the original point. At least as far as I interpreted it: k8s is not simple and you most likely didn't need it either. But if you do need a distributed PaaS, then it's probably a good idea to use it. Doesn't change the fact that it's a complex system.

Re: We were wrong about GPUs

#504

Post by SkyPuncher »

vineyardmike wrote: post #358
sgarland wrote: post #184

Earlier quoted context omitted.

Wild idea: maybe if more devs had good fundamental knowledge to begin with, the good systems engineering could be done along the way.

There is such a variety of work environments, and realistically most people learn on the job. Everyone has different skills and knowledge bases. When I was at we didn’t control our infrastructure, there were teams that did it for us. Those guys knew a lot more about the internals of Linux than your average HNer. Getting access to the SSD of the host wasn’t a sys-call away, it was a ticket to an SRE and a library impo…

This is exactly my experience. Nearly every dev on my team can dive into the details and scale that service effectively, but it’s rarely worth it.

If an engineer costs $100/hour, scaling an extra $100/month (or even an extra $1k/month) is generally a no brainer. That money is almost always better served towards shipping product.

Re: We were wrong about GPUs

#505

Post by remram »

ffsm8 wrote: post #503
remram wrote: post #496

Earlier quoted context omitted.

So if you run Kubernetes in the cloud, you consider the entire cloud provider's block storage implementation to be part of Kubernetes too? For example you'd say AWS EBS is part of Kubernetes?

In the context of this discussion, which is about the complexity of the k8s stack: yes. Youre ultimately gonna have to use a storage of some form unless you're just a stateless service/keep the services with state out of k8s. That's why I'd include it, and the fact that you can use multiple storage backends, each with their own challenges and pitfalls makes k8s indeed quiet complex. You could argue that multinode Paa…

So you're comparing Kubernetes to what? Not running services at all? In that case I agree, you're going to have to set up Linux, find a storage solution, etc as part as your setup. Then write your app. It's a lot of work.

But would I say that your entire Linux installation and the cloud it runs on is part of Kubernetes? No.

Re: We were wrong about GPUs

#506

Post by johntash »

BoorishBears wrote: post #478
Aeolun wrote: post #471

Earlier quoted context omitted.

Sure, but vast feels like renting a GPU from a rando.

Either you're familiar with GPU pricing and being willfully ignorant, or you're not familiar with the pricing in which case let someone who is point out: - "Datacenter" means it's comparable to Runpod's secure cloud pricing. - A spot instance of an H200 under someone's living room media console wouldn't go for A100 rates. $3.50 will also get you an H100 at a laundry list of providers people build real businesses on.…

You seem like you're familiar with vast. Have you used their autoscaler/serverless offering before? I haven't tried it yet, but it wasn't immediately obvious if I could have something like ollama running and scaled to zero instances when not in use.

Re: We were wrong about GPUs

#507

Post by aqueueaqueue »

dijksterhuis wrote: post #256
aqueueaqueue wrote: post #73

> developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. Is there not a market for the kind data science stuff where GPUs help but you are not using an LLM. Like statistical models on large amounts of data and so on. Maybe fly.io customer base isn't that sort of user. But I was pushing a previous company to get AWS GPUs because it would save us money vs CPU for the workload.

not from fly.io, but my experience is that most data scientists will just prefer to lump it with the tools they know (pandas / R) on CPUs, rather than delving into things like rapids https://rapids.ai -- even if it makes things faster/cheaper. I might have had a bad sample set so far. But the "doing statistics" bit seems to be the interesting thing for them. the tooling doesn't really factor into solutions/plans that…

Do many DS use Google Colab and click the GPU option? That made me think GPUs would be more popular (due to speed).

Also GPUs may be used when productionizing work done by DS but maybe I am in a tiny niche here of (Data Science) intersection (Scale up) minus (Deep learning LLM etc.)

Re: We were wrong about GPUs

#508

Post by apitman »

themgt wrote: post #421
apitman wrote: post #404

Earlier quoted context omitted.

But how many orgs need that scale?

You can run single-node k3s on a VM with 512MB of RAM and deploy your app with a hundred lines of JSON, and it inherits a ton of useful features that are managed in one place and can grow with your app if/as needed. These discussions always go in circles between Haters and Advocates: * H: "kubernetes [at planetary scale] is too complex" * A: "you can run it on a toaster and it's simpler to reason about than systemd +…

The sad thing is there probably is a toaster out there somewhere with 512MB of RAM.

Re: We were wrong about GPUs

#510

Post by ffsm8 »

remram wrote: post #505
ffsm8 wrote: post #503

Earlier quoted context omitted.

In the context of this discussion, which is about the complexity of the k8s stack: yes. Youre ultimately gonna have to use a storage of some form unless you're just a stateless service/keep the services with state out of k8s. That's why I'd include it, and the fact that you can use multiple storage backends, each with their own challenges and pitfalls makes k8s indeed quiet complex. You could argue that multinode Paa…

So you're comparing Kubernetes to what? Not running services at all? In that case I agree, you're going to have to set up Linux, find a storage solution, etc as part as your setup. Then write your app. It's a lot of work. But would I say that your entire Linux installation and the cloud it runs on is part of Kubernetes? No.

Various options around for simple alternatives, the simplest is probably just running single node.

Maybe with fail over for high availability.

Even that's fine for most deployments that aren't social media sites, aren't developed by multiple teams of devs and don't have any operations people on payroll.

Post reply on HN