Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

491–500 of 516 posts

Re: Don't rent the cloud, own instead

#491

Post by infecto »

betaby wrote: post #393
infecto wrote: post #208

Earlier quoted context omitted.

I also thinks it’s risk model too. Every time I see these kind of posts I think it misses the point there is a balance not only on cost like you describe but risk as well. You are paying to offload some of the risk from yourself.

> You are paying to offload some of the risk from yourself. The opposite is also true: one is risking being banned by exascalers.

That’s is such a tail risk that I don’t know who would actually factor it in. It has definitely happened but not with any regularity.

Re: Don't rent the cloud, own instead

#492

Post by infecto »

acdha wrote: post #359
infecto wrote: post #216

Earlier quoted context omitted.

I am not sure if you are trying to refute my lived experience or what exactly the point is. Heroku was wildly popular with startups at the time, not just those with lavish budgets. I was already touching RDS at this point and even before RDS came around no organization I worked at had me jumping on bare metal to provision services myself. There always a system in place where someone helped out engineering to deploy s…

I have no doubt that was your experience. My point was that it wasn’t even common in SV as whole, just the startup scene. Think about headcount: how many times fewer people worked at your startup than any one of Apple, Oracle, HP, Salesforce, Intuit, eBay, Yahoo, etc.? Then thing about how many other companies there are just in the Bay Area who have large IT investments even if they’re not tech companies. Even at the…

I think we are talking past each other here. While your language is a bit proactive originally your not wrong as I have already agreed startups absolutely, lavish budgets? No that’s just silly.

Again 15 years even in moderately large organizations it was quite common as a product engineer to not be responsible for the provisioning all the required services for whatever you were building. And again it’s not the rule but it is far from being an exception. Not sure what you’re trying to prove or disprove.

Re: Don't rent the cloud, own instead

#493

Post by infecto »

asimeqi wrote: post #475
infecto wrote: post #194

Earlier quoted context omitted.

It’s all anecdotal but in my experiences it’s usually opposite. Bored senior engineer wants to use something new and picks a AWS bespoke service for a new project. I am sure it happens a multitude of ways but I have never seen the case you are describing.

Since when is "bored" a synonym for "dishonest"?

Please elaborate. Such a bold statement with zero logic around it.

Re: Don't rent the cloud, own instead

#494

Post by sixothree »

user34283 wrote: post #285
JKCalhoun wrote: post #248

Naive comment from a hobbyist with nothing close to $5M: I'm curious about the degree to which you build a "home lab" equivalent. I mean if "scaling" turned out to be just adding another Raspberry Pi to the rack (where is Mr. Geerling when you need him?) I could grow my mini-cloud month by month as spending money allowed. (And it would be fun too.)

I paid 150€ for a Mini PC with an Intel N100, 16 GB of DDR5 memory, and a 500 GB SSD. While I have no intention to scale up low spec hardware like this, it at least seems to beat the Azure VMs we use at work with "4 CPUs", which corresponds to two physical cores on an AMD EPYC CPU. And that super slow machine I understand costs more than $100 per month, and that's without charges for disk space slower than the SSD, o…

It's hard to describe how slow a $150 / month azure VM really is. Holy heck are they limiting.

Re: Don't rent the cloud, own instead

#495

Post by ragall »

re-thc wrote: post #316
scalemaxx wrote: post #304

Everything comes circle. Back in my day, we just called it a "data center". Or on-premise. You know, before the cloud even existed. A 1990s VP of IT would look at this post and say, what's new? Better computing for sure. Better virtualization and administration software, definitely. Cooling and power and racks? More of the same. The argument made 2 decades ago was that you shouldn't own the infrastructure (capital ex…

> you shouldn't own the infrastructure (capital expense) and instead just account for the cost as operational expense (opex) That was part of the reason. The real reason was the internal infrastructure team in many orgs got nowhere. There was a huge queue and many teams instead had to find infinite workarounds including standing up their own. The "cloud" provided a standardized way to at least deal with this mess e.g…

> There are many places where you have to talk to a vendor with no list price

Which many places ?

Re: Don't rent the cloud, own instead

#496

Post by sanderjd »

mattbillenstein wrote: post #461
sanderjd wrote: post #436

Earlier quoted context omitted.

Absolutely yes. But you have to do this either way. So it's just purely additive work to run the infrastructure as well. I think if it were true that the tuning is easier if you run the infrastructure yourself, then this would be a good point. But in my experience, this isn't the case for a couple reasons. First of all, the majority of tuning wins (indexes, etc.) are not on the infrastructure side, so it's not a big…

Maybe, but you're paying through the nose continually for something you could learn to do once - or someone on your team could easily learn with a little time and practice. Like, if this is a tradeoff you want to make, it's fine, but at some point learning that 10% more can halve your hosting costs so it's well worth it.

It's not the learning, it's the ongoing commitment of time and energy into something that is not a differentiator for the business (unless it is actually a database hosting business).

I can see how the cost savings could justify that, but I think it makes sense to bias toward avoiding investing in things that are not related to the core competency of the business.

Re: Don't rent the cloud, own instead

#497

Post by sanderjd »

Symbiote wrote: post #486
sanderjd wrote: post #407

Earlier quoted context omitted.

No not at all, I have the same opinion as you! But I'm curious to understand the opposite view.

I ran through roughly our numbers here [1], it looks like self-hosted costs us about 25% of AWS. I didn't include labour costs, but the self-hosted tasks (set up of hardware, OS, DB, backup, monitoring, replacing a failed component which would be really unusual) are small compared to the labour costs of the DB generally (optimizing indices, moving data around for testing etc, restoring from a backup). [1] https://new…

Yes thank you for that. I always feel like these up front cost analyses miss (or underrate) the ongoing operational cost to monitor and be on call to fix infrastructure when problems occur. But I do find it plausible that the cost savings are such that this can be a wise choice nonetheless.

Re: Don't rent the cloud, own instead

#498

Post by tracker1 »

sgarland wrote: post #415
tracker1 wrote: post #360

Earlier quoted context omitted.

>> vs. relying on a new backend instance per client in a cloud infrastructure > You want to spin up an EC2 per client, and run an isolated copy of the application, isolated DB, etc. inside of it? That sounds like a nightmare to manage, especially if you want or need HA capabilities. No... just running a new hosted database instance per client... but (re)using your service/application infrastructure, but just connecti…

That’s going to be enormously expensive. If you need guaranteed tenant isolation, put them in separate schemas, with specific user grants. That scales up much better than you’d think. Or use Citus Postgres, and get sharding by schema for free, so you have both isolation and more or less infinite growth. I’m not sure why if you think it would take 5 employees to manage self-hosted DBs, that it won’t take close to that…

Expensive how? It should be baked into your contract cost per client... maybe more expensive than you can run at scale.

As to 5 dedicated employees for db systems management... that's just roughly where I would put the breakpoint... short of that you're more likely to have people in mixed roles during development, where people spend only part of their time managing migrations for schema changes and most of their time will be developing features.

The schema, table, index design etc. are largely done by the developers themselves at a startup level.... and even then, it's a problem where the costs can be op-ex against direct revenue scaling. So having 1000 clients isn't relatively more or less expensive than the first 5-10, it's baked into the model.

Re: Don't rent the cloud, own instead

#499

Post by eitally »

toomuchtodo wrote: post #269
organsnyder wrote: post #263

Earlier quoted context omitted.

I've worked with many "enterprise IT" sysadmins (in healthcare, specifically). Some are very proficient generalists, but most (in my experience) are fluent in only their specific platforms, no different than the typical AWS engineer.

Perhaps we need bootcamps for on prem stacks if we are concerned about a skills gap. This is no different imho from the trades skills shortage many developed countries face. The muscle must be flexed. Otherwise, you will be held captive by a provider "who does it all for you". "Today, we are going to calculate the power requirements for this rack, rack the equipment, wire power and network up, and learn how to use PX…

I don't really think so. That was a ship that sailed ten years ago and nearly every sysadmin who is still proficient with managing on-prem stacks has adapted to also learn how to manage VPCs in an arbitrary cloud. It's not like this is a recent change.

Re: Don't rent the cloud, own instead

#500

Post by antonvs »

rcxdude wrote: post #489
antonvs wrote: post #315

Earlier quoted context omitted.

> One of AWS's favorite situations I'll give you an alternative scenario, which IME is more realistic. I'm a software developer, and I've worked at several companies, big and small and in-between, with poor to abysmal IT/operations. I've introduced and/or advocated cloud at all of them. The idea that it's "more expensive" is nonsense in these situations. Calculate the cost of the IT/operations incompetence, and the c…

Fron what I've seen this can work as a stopgap until IT get their hooks into the cloud system in which case you circle back to paying to costs of incompetence and the costs of the cloud (sometimes stacking on top of each other).

There's still a benefit in terms of infrastructure reliability. Recovery times are faster, backups more reliable, etc. Basically, vendor managed is better than customer managed in most situations, assuming a competent vendor.

Also, if the cloud systems are architected properly before IT gets hold of them, then they tend to retain their good properties for a long time, especially if others are paying attention to e.g. gitops pull requests.

My current company ended up replacing its (small) operations team in order to get people with cloud expertise. We hired the new team for the skills we needed. It's worked out well.

Post reply on HN