Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

431–440 of 516 posts

Re: Don't rent the cloud, own instead

#431

Post by dev_l1x_be »

objektif wrote: post #203
Lucasoato wrote: post #95

Earlier quoted context omitted.

Hetzner is definitely an interesting option. I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) but the price difference makes it so appealing. From our financial models, Hetzner can win over AWS when you spend over 10~15K per month on infrastructure and you’re hiring really well. It’s still a risk, but a risk that definitely can be worthy.

No amount of money will make me maintain my own dbs. We tried it at first and it was a nightmare.

Or CDN, queues, log service, observability, distributed storage. I am not even sure what the people in the on-prem vs cloud argument think. If you need a highly specialised infra with one or two core services and a lower tier network is ok then on-prem is ok. Otherwise if is a never ending quest to re-discover the millions of engineering hours went into building something like AWS.

Re: Don't rent the cloud, own instead

#432

Post by mattbillenstein »

rockwotj wrote: post #147
baby wrote: post #143

Earlier quoted context omitted.

I’m wondering if it makes sense to distribute your architecture so that workers who do most of the heavy lifting are in hetzner, while the other stuff is in costly AWS. On the other hand this means you don’t have easy access to S3, etc.

networking costs are so high in AWS I doubt this makes sense

Depends on how data-heavy the work is. We run a bunch of gpu training jobs on other clouds with the data ending up in S3 - the extra transfer costs wrt what we save on getting the gpus from the cheapest cloud available, it makes a lot of sense.

Also, just availability of these things on AWS has been a real pain - I think every startup got a lot of credits there, so flood of people trying to then use them.

Re: Don't rent the cloud, own instead

#433

Post by mattbillenstein »

sanderjd wrote: post #405
cheema33 wrote: post #381

Earlier quoted context omitted.

> Managing the PostgreSQL databases is a medium to low complexity task as I see it. Same here. But, I assume you have managed PostgreSQL in the past. I have. There are a large number of people software devs who have not. For them, it is not a low complexity task. And I can understand that. I am a software dev for our small org and I run the servers and services we need. I use ansible and terraform to automate as much…

For what it's worth, I have also managed my own databases, but that's exactly why I don't think it's a good use of my time. Because it does take time! And managed database options are abundant, inexpensive, and perform well. So I just don't really see the appeal of putting time into this.

If you have a database, you still have work to do - optimizing, understanding indexes, etc. Managed services don't solve these problems for you magically and once you do them, just running the db itself isn't such a big deal and it's probably easier to tune for what you want to do.

Re: Don't rent the cloud, own instead

#434

Post by Torq_boi »

ActorNightly wrote: post #427
jillesvangurp wrote: post #29

At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. Which means it doesn't make much sense for the majority of startup companies until they become late stage and their hosting cost actually becomes a big cost burden. There are in between solutions. Renting bare metal instead of renting virtual machines can be quite n…

>At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. The issue with comma.ai is that the company is HEAVILY burdened with Geohotz ideals, despite him no longer even being on the board. I used to be very much into his streams and he rants about it plenty. A large reason of why they run their own datacenter is that the…

Our preference for training in our own datacenter has nothing to do with wokeness. Did you read the blog post? The reasons are clearly explained.

The climate in Wyoming and Montana are actually worse in terms of climate. San Diego's climate extremes are less extreme than those places. Though moving out of CA is a good idea for power cost reasons, also addressed in the blog.

Re: Don't rent the cloud, own instead

#435

Post by lstodd »

muvlon wrote: post #101
bpavuk wrote: post #83

Earlier quoted context omitted.

if someone on the DevOps team knows Nix, option 3 becomes a lot cheaper time-wise! yeah, Nix flakes still need maintenance, especially on the `nixos-unstable` branch, but you get the quickest disaster recovery route possible! plus, infra flexibility removes random constraints that e.g. Cloudflare Workers have

Option 4 as well, that's how we do it at work and it's been great. However, it can't really be "someone on the team knows Nix", anyone working on Ops will need Nix skills in order to be effective.

Why this fixation on Nix? You don't need Nix to run bare metal.

Re: Don't rent the cloud, own instead

#436

Post by sanderjd »

mattbillenstein wrote: post #433
sanderjd wrote: post #405

Earlier quoted context omitted.

For what it's worth, I have also managed my own databases, but that's exactly why I don't think it's a good use of my time. Because it does take time! And managed database options are abundant, inexpensive, and perform well. So I just don't really see the appeal of putting time into this.

If you have a database, you still have work to do - optimizing, understanding indexes, etc. Managed services don't solve these problems for you magically and once you do them, just running the db itself isn't such a big deal and it's probably easier to tune for what you want to do.

Absolutely yes. But you have to do this either way. So it's just purely additive work to run the infrastructure as well.

I think if it were true that the tuning is easier if you run the infrastructure yourself, then this would be a good point. But in my experience, this isn't the case for a couple reasons. First of all, the majority of tuning wins (indexes, etc.) are not on the infrastructure side, so it's not a big win to run it yourself. But then also, the professionals working at a managed DB vendor are better at doing the kind of tuning that is useful on the infra side.

Re: Don't rent the cloud, own instead

#437

Post by lstodd »

speedgoose wrote: post #49

I would suggest to use both on-premise hardware and cloud computing. Which is probably what comma is doing. For critical infrastructure, I would rather pay a competent cloud provider than being responsible for reliability issues. Maintaining one server room in the headquarters is something, but two servers rooms in different locations, with resilient power and network is a bit too much effort IMHO. For running many s…

I don't get why most everyone insists on comparing cloud to on-premises and not to dedicated. Why would anyone run own DC infra when there's Hetzner and many others?

Re: Don't rent the cloud, own instead

#438

Post by lstodd »

riku_iki wrote: post #416
epolanski wrote: post #90

Earlier quoted context omitted.

> but two servers rooms in different locations, with resilient power and network is a bit too much effort IMHO I worked in a company with two server farms (a main and a a backup one essentially) in Italy located in two different regions and we had a total of 5 employees taking care of them. We didn't hear about them, we didn't know their names, but we had almost 100% uptime and terrific performance. There was one sin…

it depends on complexity of your infra. Running full scale kubernets, with multiple databases and services and expected 99.99% uptime likely can't be handled by one person.

Takes a team of 3-4 in my experience. One person doesn't cut it when the talk of percents of uptime starts no matter what scale. (and no matter cloud, dedicated or on-premises).

Re: Don't rent the cloud, own instead

#439

Post by CamperBob2 »

hbogert wrote: post #9

Datacenters need cool dry air? No, low isn't good perse. I worked in a datacenter which in winters had less than 40%, ram was failing all over the place. Low humidity causes static electricity.

Low humidity causes static electricity.

RAM that is plugged in and operating isn't subject to external ESD, unless you count lightning strikes. Where are you getting this?

Re: Don't rent the cloud, own instead

#440

Post by dh2022 »

LOL’ed IRL at “ In a future blog post I hope I can tell you about how we produce our own power and you should too.” Producing own power as a pre-requisite for running on-prem is a non-starter for many.
Post reply on HN