Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

341–350 of 516 posts

Re: Don't rent the cloud, own instead

#341

Post by sanderjd »

Symbiote wrote: post #309
sanderjd wrote: post #278

Earlier quoted context omitted.

Is it though? This is a genuine question. My intuition is that the investment of time / stress / risk to become good at this is unlikely to have high ROI to either the person putting in that time or to the business paying them to do so. But maybe that's not right.

Managing the PostgreSQL databases is a medium to low complexity task as I see it. Take two equivalent machines, set up with streaming replication exactly as described in the documentation, add Bacula for backups to an off-site location for point-in-time recovery. We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'…

This sounds medium to high complexity to me. You need to do all those things, and also have multiple people who know how to do them, and also make sure that you don't lose all the people who know how to do them, and have one of those people on call to be able to troubleshoot and fix things if they go wrong, and have processes around all that. (At least if you are running in production with real customers depending on you, you should have all those things.)

With a managed solution, all of that is amortized into your monthly payment, and you're sharing the cost of it across all the customers of the provider of the managed offering.

Personally, I would rather focus on things that are in or at least closer to the core competency of our business, and hire out this kind of thing.

Re: Don't rent the cloud, own instead

#342

Post by rob_c »

And finally we reach the point where you're not shot for explaining if you invest in ownership after everything is over you have something left that has intrinsic value regardless of what you were doing with it.

Otherwise, well just like that gym membership, you get out what you put into it...

Re: Don't rent the cloud, own instead

#343

Post by unethical_ban »

infecto wrote: post #200
iso1631 wrote: post #140

Earlier quoted context omitted.

> I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) Out of interest, how old are you? This was quite normal expectation of a technical department even 15 years ago.

No it was not. 15 years ago Heroku was the rage. Even the places that had bare metal usually had someone running something similar to devops and at least core infrar was not being touched. I am sure places existed but 15 years while far away was already pretty far along from what you describe. At least in SV.

SV and financial services are quite different.

It's 2026 and banks are still running their mainframe, running windows VMs on VMware and building their enterprise software with Java.

The big boys still have their own datacenters they own.

Sure, they try dabbling with cloud services, and maybe they've pushed their edge out there, and some minor services they can afford to experiment with.

Re: Don't rent the cloud, own instead

#344

Post by iso1631 »

infecto wrote: post #216
acdha wrote: post #206

Earlier quoted context omitted.

Heroku was popular with startups who didn’t have infrastructure skills but the price was high enough that anyone who wasn’t in that triangle of “lavish budget, small team, limited app diversity” wasn’t using it. Things like AWS IaaS were far more popular due to the lower cost and greater flexibility but even that was far from a majority service class.

I am not sure if you are trying to refute my lived experience or what exactly the point is. Heroku was wildly popular with startups at the time, not just those with lavish budgets. I was already touching RDS at this point and even before RDS came around no organization I worked at had me jumping on bare metal to provision services myself. There always a system in place where someone helped out engineering to deploy s…

> Heroku was wildly popular with startups

The world's a lot bigger than startups

Re: Don't rent the cloud, own instead

#345

Post by throwawaypath »

dagi3d wrote: post #130

> San Diego power cost is over 40c/kWh, ~3x the global average. It’s a ripoff, and overpriced simply due to political dysfunction. Mind anyone elaborate? Always thought this is was a direct cause of the free market. Not sure if by dysfunction the op means lack of intervention.

>Mind anyone elaborate? Always thought this is was a direct cause of the free market. Not sure if by dysfunction the op means lack of intervention.

The majority of Californians have no say and cannot choose their utilities provider. This is the polar opposite of the "free market".

Re: Don't rent the cloud, own instead

#346

Post by lumost »

torginus wrote: post #151
adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

I think the issue with this formulation is what drives the cost at cloud providers isn't necessarily that their hardware is too expensive (which it is), but that they push you towards overcomplicated and inefficient architectures that cost too much to run. A core at this are all the 'managed' services - if you have a server box, its in your financial interest to squeeze as much per out of it as possible. If you're us…

I don’t understand why most cloud backend designs seem to strive for maximizing the number of services used.

My biggest gripe with this is async tasks where the app does numerous hijinks to avoid a 10 minute lambda processing timeout. Rather than structure the process to process many independent and small batches, or simply using a modest container to do the job in a single shot - a myriad of intermediate steps are introduced to write data to dynamo/s3/kinesis + sqs/and coordination.

A dynamically provisioned, serverless container with 24 cores and 64 GB of memory can happily process GBs of data transformations.

Re: Don't rent the cloud, own instead

#347

Post by sgarland »

tracker1 wrote: post #330
sgarland wrote: post #317

Earlier quoted context omitted.

> But until then it's a long term cost optimization with really high upfront capital expenditure and risk. The upfront capex does not need to be that high, unless you're running your own AI models. Other than leasing new ones, as a sibling comment stated, you can buy used. You can get a solid Dell 2U with a full service contract (3 years) for ~$5-10K depending on CPU / memory / storage configuration. Or if you don't…

And if each of your clients is in the Healthcare industry and dealing with end-user medical data? Or financial data? Are you prepared for appropriate data isolation/sharding and controls? Do you have a strategy for scaling database operations per client or across all clients? It really depends on the business model as to how well you might support your own infrastructure vs. relying on a new backend instance per clie…

> And if each of your clients is in the Healthcare industry and dealing with end-user medical data? Or financial data?

Then you're probably going to need some combination of HIPAA / SOC 2 / PCI DSS certification, regardless of where your servers are physically located. AWS has certified the infrastructure side for you, but that doesn't remove your obligations for the logical side.

> Are you prepared for appropriate data isolation/sharding and controls? Do you have a strategy for scaling database operations per client or across all clients?

Again, you're going to need that regardless of where your servers physically exist.

> vs. relying on a new backend instance per client in a cloud infrastructure

You want to spin up an EC2 per client, and run an isolated copy of the application, isolated DB, etc. inside of it? That sounds like a nightmare to manage, especially if you want or need HA capabilities.

Re: Don't rent the cloud, own instead

#348

Post by ex-aws-dude »

I can see how this would work fine if the primary purpose is for training rather than serving large volumes of customer traffic in multiple regions

It would probably even make sense for some companies to still use cloud for their API but do the training on prem as that may be the expensive part.

Re: Don't rent the cloud, own instead

#349

Post by [deleted] »

kevinkatzke wrote: post #276

Feels like I’ve lived through a full infrastructure fashion cycle already. I started my career when cloud was the obvious answer and on-prem was “legacy.” Now on-prem is cool again. Makes me wonder whether we’re already setting up the next cycle 10 years from now, when everyone rediscovers why cloud was attractive in the first place and starts saying “on-prem is a bad idea” again.

[deleted]

Re: Don't rent the cloud, own instead

#350

Post by sanderjd »

adamcharnock wrote: post #336
sanderjd wrote: post #321

Earlier quoted context omitted.

This space of #2 like Lithus is not something I'm very familiar with, so thank you for the comment that piqued my interest! If you're willing to share, I'm curious who else you would describe as being in this space. My last decade and a half or so of experience has all been in cloud services, and prior to that it was #3 or #4. What was striking to me when I went to the Lithus website was that I couldn't figure out an…

Hello! I think this is a fair question, and improving the communication on the website is something that is steadily climbing up our priority list. We're not really that kind of product company; we're more of a services company. What we do is deploy Kubernetes clusters onto bare metal servers. That's the core technical offering. However, everything beyond that is somewhat per-client. Some clients need a lot of comput…

This makes total sense to me. I'm thinking through the flow that would lead me to be a customer of yours.

In my current job, I think we're honestly a bit past the phase where I would want to take on a migration to a service like yours. We already have a good team of infrastructure folks running our cloud infrastructure, and we have accepted the lock-in of various AWS managed services. So the high-touch devops support doesn't sound that useful to me (we already have people who are good at this), and replacing all the locked-in components seems unlikely to have good ROI. I think we'd be more likely to go straight to #3 if we decided to take that on to save money.

But I'll probably be a founder or early employee at a new startup again someday, and I'm intrigued by your offering from that perspective. But it seems pretty clear to me that I shouldn't call you up on day 1, because I'm going to be nowhere near $5k a month, and I want to move faster than calling someone up to talk about my needs. I want to self-serve a small amount of usage, and cloud services seem really great for that. But this is how they get you! Once you've started with a particular cloud service, it's always easiest to take on more lock-in.

At some point between these two situations, though, I can see where your offering would be great. But the decision point isn't all that clear to me. In my experience, by the time you start looking at your AWS bill and thinking "crap, that seems pretty expensive", you have better things to do than an infrastructure migration, and you have taken on some lock-in.

I do like the idea of high-touch services to solve the breaking-the-lock-in challenge! I'll certainly keep this in mind next time I find myself in this middle ground where the cloud starts feeling more expensive than it's worth, but we don't want to go straight to #3.

Post reply on HN