Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

461–470 of 604 posts

Re: We were wrong about GPUs

#461

Post by sgarland »

ketzo wrote: post #295
godelski wrote: post #293

Earlier quoted context omitted.

> We can wish everyone were good at everything, or we can try to actually get things done. False dichotomy. There's no reason we can't have both. I want to be clear, there's no perfect code or a perfect understanding or any of that. But the complaint here about not knowing /enough/ fundamentals is valid. There is some threshold which we should recognize as a minimum. The disagreement is about where this threshold is,…

It’s not a false dichotomy at all. You only have so many hours in a day. At a startup, it’s very unlikely (certainly not impossible!) that your differentiation will come from very cheap system orchestration - your time is likely better spent on building your product. Minimum knowledge is one thing; minimum time to apply it is another.

If you had to spend time / VC money learning all of this stuff before you could begin to apply it, I absolutely agree, it's a waste of time. That's not my point. My point is people (by people, I mean "someone interested in tech and is likely to pursue it as a career") can and should learn these things earlier in life such that it's trivial once they're in the workforce.

I could go from servers sitting on the ground to racked, imaged, and ready to serve traffic in a few hours, because I've spent the time learning how to do it, and have built scripts and playbooks to do so. Even if I hadn't done the latter, many others have also done so and published them, so as long as you knew what you were looking for, you could do the same.

Re: We were wrong about GPUs

#462

Post by kbolino »

sgarland wrote: post #454
kbolino wrote: post #394

Earlier quoted context omitted.

For $200/month, I can have 2 ALBs, 2 ECS services, 2 CloudWatch log groups, and 2 RDS instances on AWS (one each for dev and prod) and a GitHub Team account with enough included runner minutes to cover most deployments. A colo is going to be more hassle, and I'll have to monitor more things (like system upgrades and intrusion attempts). I'd also have to amortize parts and labor as part of the cost, which is going to…

A Postgres `db.m6g.large` (the cheapest non-burstable instance) runs $114/month for a single AZ, and that's not counting storage or bandwidth. A `db.t4g.medium` runs $47/month, again, not counting storage or bandwidth. An ALB that somehow only consumed a single LCU per month would run $22. The rest of the mentioned items will vary wildly depending on the application, but between those and bandwidth - not to mention G…

I upvoted for the details, and I agree that if you try to buy comparable capacity in AWS, it's going to be more expensive. Scaling up in AWS is definitely going to cost more over time too. I don't want to hide these facts.

But you don't need comparable capacity, at least not at first. And when you do, you click some buttons or run terraform plan/apply. Absolutely it's going to cost more measured only by tech specs. But you're not paying primarily for tech specs, you're paying for somebody else to do the work. That's where the cost comparability really needs to be assessed.

Security in AWS is a thorny topic, I'll agree, but the risks are a little different. You need to secure your accounts and users, and lock out unneeded services while monitoring for unexpected service utilization. Honestly, I think for what you're paying, AWS should be doing more for you here (and they are improving albeit slowly). Hence maybe the real point of comparison ought to be against PaaS because then all of that is out of scope too, and I think such offerings are already putting pressure on AWS to offer more value.

Re: We were wrong about GPUs

#463

Post by sgarland »

chromanoid wrote: post #415
sgarland wrote: post #184

Earlier quoted context omitted.

Wild idea: maybe if more devs had good fundamental knowledge to begin with, the good systems engineering could be done along the way.

Premature optimization may hit them hard. Overengineering is imo usually the bigger technical debt and a huge upfront cost as well. Well-thought out plans tend to become a sunken cost fallacy. Making room for changes is hard enough in XP like ways of working. When you have to tell your manager that half a year of careful plans and engineering can be thrown away, because of the new requirements, which emerge from late…

Infra should not require much in the way of redoing if it's done correctly. Foundational software's configuration like RDBMS schema, maybe, but I wouldn't classify that as infra per se.

Seriously, I'm struggling to figure out how "we have servers that run containers / applications" would need to be redone just because the application changed.

Re: We were wrong about GPUs

#464

Post by mlhpdx »

sgarland wrote: post #454
kbolino wrote: post #394

Earlier quoted context omitted.

For $200/month, I can have 2 ALBs, 2 ECS services, 2 CloudWatch log groups, and 2 RDS instances on AWS (one each for dev and prod) and a GitHub Team account with enough included runner minutes to cover most deployments. A colo is going to be more hassle, and I'll have to monitor more things (like system upgrades and intrusion attempts). I'd also have to amortize parts and labor as part of the cost, which is going to…

A Postgres `db.m6g.large` (the cheapest non-burstable instance) runs $114/month for a single AZ, and that's not counting storage or bandwidth. A `db.t4g.medium` runs $47/month, again, not counting storage or bandwidth. An ALB that somehow only consumed a single LCU per month would run $22. The rest of the mentioned items will vary wildly depending on the application, but between those and bandwidth - not to mention G…

The heartburn I have is how this stifles innovation for some. The cost of experimentation is high if every person wanting to try a new idea goes down this road.

My personal AWS account is stuffed with globally distributed multi-region, multi-az, fault tolerant, hugely scalable things that rarely get used. By “rarely” I mean requests per hour or minute, not second.

The sum total CPU utilization would be negligible. And if I ran instances across the 30+ AZs I’d be broke.

The service based approach (aka event driven) has some real magic at the low end of usage where experimentation and innovation happens.

Re: We were wrong about GPUs

#465

Post by sgarland »

kbolino wrote: post #462
sgarland wrote: post #454

Earlier quoted context omitted.

A Postgres `db.m6g.large` (the cheapest non-burstable instance) runs $114/month for a single AZ, and that's not counting storage or bandwidth. A `db.t4g.medium` runs $47/month, again, not counting storage or bandwidth. An ALB that somehow only consumed a single LCU per month would run $22. The rest of the mentioned items will vary wildly depending on the application, but between those and bandwidth - not to mention G…

I upvoted for the details, and I agree that if you try to buy comparable capacity in AWS, it's going to be more expensive. Scaling up in AWS is definitely going to cost more over time too. I don't want to hide these facts. But you don't need comparable capacity, at least not at first. And when you do, you click some buttons or run terraform plan/apply. Absolutely it's going to cost more measured only by tech specs. B…

> But you don't need comparable capacity, at least not at first.

Agreed.

> But you're not paying primarily for tech specs, you're paying for somebody else to do the work. ... Honestly, I think for what you're paying, AWS should be doing more for you here

Also agreed, and this is why I don't think the value proposition exists.

We can agree to disagree on which approach is better; I doubt there's an objective truth to be had.

Re: We were wrong about GPUs

#466

Post by sota_pop »

morphle wrote: post #263
digdugdirk wrote: post #258

Earlier quoted context omitted.

Interesting! Fingers crossed someone who's looking for your skillset finds your post. What is your process to turn Mac minis into a cluster? Is there any special hardware involved? And you can get 100x tok/s vs others on comparable hardware, what do you do differently - hardware, software, something else?

I discussed all the points you ask about in my HN postings last month, but never in enough detail so you must ask me to specify and that's when people hire me. As you can see from this comments thread, most people, especially programmers, lack the knowledge we computer scientist, parallel programmers and chip or hardware designers have. >What is your process Science. To measure is to know, my prof always said. To ans…

Sounds interesting, but I don’t see any HN submissions on your profile last month. Are you referring to comments you made?

Re: We were wrong about GPUs

#467

Post by chasd00 »

zeroq wrote: post #308
never_inline wrote: post #296

Earlier quoted context omitted.

Core kubernetes (deployments, services etc..) is fairly easy to understand. lot of other stuff in the cncf ecosystem is immature. I don't think most people need to use all the operators, admission controllers, otel, service mesh though. If you're running one team with all services trusting each other, you don't have problems solved by these things. Whenever you introduce a CNCF component outside core kubernetes, inve…

Having a team that runs the kubernetes for you and being on receiving end is indeed super easy. Need another microservice? Just add another repository, add short yaml, push it to CI and bam!, it's online. But setting it up is not a trivial task and often a recipe for disaster. I've seen a fair share of startups who took too much kool aid and wanted parrot FANG stacks just to discover they are burning tons of money ju…

The irony is the whole devops and cloud sales pitch was developers can do all this themselves and you no longer need an sysadmin team. Turns out you still do, it’s just called the devops/cloud team and not sys admin team.

Re: We were wrong about GPUs

#468

Post by sudhirj »

freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

This is context based dichotomy, not a person-based one.

In my personal life, I’m curiosity-oriented, so I put my blog, side projects and mom’s chocolate shop on fully self hosted VPSs.

At my job managing a team of 25 and servicing thousands of customers for millions in revenue, I’m very results-oriented. Anyone who tries to put a single line of code outside of a managed AWS service is going to be in a lot of trouble with me. In a results-oriented environment, I’m outsourcing a lot of devops work to AWS, and choosing to pay a premium because I need to use the people I hire to work on customer problems.

Trying to conflate the two orientations with mindsets / personality / experience levels is inaccurate. It’s all about context.

Re: We were wrong about GPUs

#469

Post by DrFalkyn »

formerly_proven wrote: post #443
remram wrote: post #414

Earlier quoted context omitted.

CSI and CNI do about as much magic as `docker volume` and `docker network`. People act like their web framework and SQL connection pooler and stuff are so simple, while Kubernetes is complex and totally inscrutable for mortals, and I don't get it. It has a couple of moving parts, but it is probably simpler overall than SystemD.

I was genuinely surprised that k8s turned out to actually be pretty straightforward and very sensible after years of never having anything to do with it and just hearing about it on the net. Turns out opinions are just like after all. That being said, what people tend to build on top of that foundation is a somewhat different story.

it’s not k8s. It’s distrusted systems

Unfortunately people (cough managers) think k8s is some magic that makes distrusted systems problems go away, and automagically enables unlimited scalability

In reality it just makes the mechanics a little easier and centralized

Getting distributed systems right is usually difficult

Re: We were wrong about GPUs

#470

Post by ChoHag »

serviceberry wrote: post #101
_dark_matter_ wrote: post #74

Earlier quoted context omitted.

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…

> The truth is we are constantly moving further and further away from the silicon. Are we? We're constantly changing abstractions, but we don't keep adding them all that often. Operating systems and high-level programming languages emerged in the 1960s. Since then, the only fundamentally new layer of abstraction were virtual machines (JVM, browser JS, hardware virtualization, etc). There's still plenty of hardware-sp…

[dead]
Post reply on HN