Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

411–420 of 516 posts

Re: Don't rent the cloud, own instead

#411

Post by doctorpangloss »

adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

Where do AWS reserved instances come into your hierarchy? What if there existed a “perpetual” reserved instance? Is cap-ex vs. op-ex really the key distinction?

Re: Don't rent the cloud, own instead

#412

Post by Dylan16807 »

cheema33 wrote: post #381
Symbiote wrote: post #309

Earlier quoted context omitted.

Managing the PostgreSQL databases is a medium to low complexity task as I see it. Take two equivalent machines, set up with streaming replication exactly as described in the documentation, add Bacula for backups to an off-site location for point-in-time recovery. We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'…

> Managing the PostgreSQL databases is a medium to low complexity task as I see it. Same here. But, I assume you have managed PostgreSQL in the past. I have. There are a large number of people software devs who have not. For them, it is not a low complexity task. And I can understand that. I am a software dev for our small org and I run the servers and services we need. I use ansible and terraform to automate as much…

> Same here. But, I assume you have managed PostgreSQL in the past. I have. There are a large number of people software devs who have not. For them, it is not a low complexity task. And I can understand that.

I'd say needing to read the documentation for the first time is what bumps it up from low complexity to medium. And then at medium you should still do it if there's a significant cost difference.

Re: Don't rent the cloud, own instead

#413

Post by objektif »

cheema33 wrote: post #381
Symbiote wrote: post #309

Earlier quoted context omitted.

Managing the PostgreSQL databases is a medium to low complexity task as I see it. Take two equivalent machines, set up with streaming replication exactly as described in the documentation, add Bacula for backups to an off-site location for point-in-time recovery. We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'…

> Managing the PostgreSQL databases is a medium to low complexity task as I see it. Same here. But, I assume you have managed PostgreSQL in the past. I have. There are a large number of people software devs who have not. For them, it is not a low complexity task. And I can understand that. I am a software dev for our small org and I run the servers and services we need. I use ansible and terraform to automate as much…

How do you manage availability zones in your fully self managed setup?

Re: Don't rent the cloud, own instead

#414

Post by strobe »

Aurornis wrote: post #299
bojangleslover wrote: post #179

Earlier quoted context omitted.

The complexity is what gets you. One of AWS's favorite situations is 1) Senior engineer starts on AWS 2) Senior engineer leaves because our industry does not value longevity or loyalty at all whatsoever (not saying it should, just observing that it doesn't) 3) New engineer comes in and panics 4) Ends up using a "managed service" to relieve the panic 5) New engineer leaves 6) Second new engineer comes in and not only…

One of the biggest problems with the self-hosted situations I’ve seen is when the senior engineers who set it up leave and the next generation has to figure out how to run it all. In theory with perfect documentation they’d have a good head start to learn it, but there is always a lot of unwritten knowledge involved in managing an inherited setup. With AWS the knowledge is at least transferable and you can find peopl…

>With AWS the knowledge is at least transferable

unfortunately it lot of things in AWS that also could be messed up so it might be really hard to research what is going on. For example, you could have hundreds of Lambdas running without any idea where original sources and how they connected to each-other, or complex VPCs network routing where some rules and security groups shared randomly between services so if you do small change it could lead to completely difference service to degrade (like you were hired to help with service X but after you changes some service Y went down and you even not aware that it existed)

Re: Don't rent the cloud, own instead

#415

Post by sgarland »

tracker1 wrote: post #360
sgarland wrote: post #347

Earlier quoted context omitted.

> And if each of your clients is in the Healthcare industry and dealing with end-user medical data? Or financial data? Then you're probably going to need some combination of HIPAA / SOC 2 / PCI DSS certification, regardless of where your servers are physically located. AWS has certified the infrastructure side for you, but that doesn't remove your obligations for the logical side. > Are you prepared for appropriate d…

>> vs. relying on a new backend instance per client in a cloud infrastructure > You want to spin up an EC2 per client, and run an isolated copy of the application, isolated DB, etc. inside of it? That sounds like a nightmare to manage, especially if you want or need HA capabilities. No... just running a new hosted database instance per client... but (re)using your service/application infrastructure, but just connecti…

That’s going to be enormously expensive. If you need guaranteed tenant isolation, put them in separate schemas, with specific user grants. That scales up much better than you’d think.

Or use Citus Postgres, and get sharding by schema for free, so you have both isolation and more or less infinite growth.

I’m not sure why if you think it would take 5 employees to manage self-hosted DBs, that it won’t take close to that to manage cloud-hosted ones. The only real difference you’re going to have once both are set up is dealing with any possible hardware issues. The initial setup for backups, streaming replication, etc. is a one-time thing, and then it just works. Hire a contractor for that, optionally keeping them on retainer for emergencies if you want.

You still have to deal with DB issues with a managed service: things like schema management, table design, index maintenance, parameter tuning, query optimization are all your responsibility, not the cloud provider’s.

Re: Don't rent the cloud, own instead

#416

Post by riku_iki »

epolanski wrote: post #90
speedgoose wrote: post #49

I would suggest to use both on-premise hardware and cloud computing. Which is probably what comma is doing. For critical infrastructure, I would rather pay a competent cloud provider than being responsible for reliability issues. Maintaining one server room in the headquarters is something, but two servers rooms in different locations, with resilient power and network is a bit too much effort IMHO. For running many s…

> but two servers rooms in different locations, with resilient power and network is a bit too much effort IMHO I worked in a company with two server farms (a main and a a backup one essentially) in Italy located in two different regions and we had a total of 5 employees taking care of them. We didn't hear about them, we didn't know their names, but we had almost 100% uptime and terrific performance. There was one sin…

it depends on complexity of your infra.

Running full scale kubernets, with multiple databases and services and expected 99.99% uptime likely can't be handled by one person.

Re: Don't rent the cloud, own instead

#417

Post by iso1631 »

bluGill wrote: post #364
cheema33 wrote: post #361

Earlier quoted context omitted.

> AWS has people working 24x7 so that if something fails someone is there to take action.. The number of things that these 24x7 people from AWS will cover for you is small. If your application craps out for any number of reasons that doesn't have anything to do with AWS, that is on you. If your app needs to run 24x7 and it is critical, then you need your own 24x7 person anyway.

All the hardware and network issues are on them. I agree that you still need your own people to support you applications, but that is only part of the problem.

I've got thousands of devices over hundreds of sites in dozens of countries. The number of hardware failures are a tiny number, and certainly don't need 24/7 response

Meanwhile AWS breaks once or twice a year.

Re: Don't rent the cloud, own instead

#418

Post by boplicity »

cheema33 wrote: post #369
boplicity wrote: post #258

Earlier quoted context omitted.

It's a server with: - 2x Intel Xeon 5218 - 128gb Ram - 2x960GB SSD - 30TB monthly bandwidth I pay around an extra $200/month for "premium" support and Acronis backups, both of which have come in handy, but are probably not necessary. (Automated backups to AWS are actually pretty cheap.) It definitely helps with peace of mind, though.

I have a similar system from Hetzner. I pay around $100 for it. No bandwidth cap. I have setup encrypted backups to go to my backup server in the office. We have a gigabit service at the office. Critical data changes are backed up every hour and full backup once a day.

Yeah -- I know I could probably get a better deal. I pay more for premium support ($200), as well as a North American location. Plus, probably an addition premium for not wanting to go through the effort of switching servers.

Re: Don't rent the cloud, own instead

#419

Post by phailhaus »

tgtweak wrote: post #385

>San Diego has a mild climate and we opted for pure outside air cooling. This gives us less control of the temperature and humidity, but uses only a couple dozen kW. We have dual 48” intake fans and dual 48” exhaust fans to keep the air cool. To ensure low humidity ( Oh man, this is bad advice. Airborn humidity and contaminants will KILL your servers on a very short horizon in most places - even San Diego. I highly s…

I didn't even know this is something you had to worry about. This is why I use the cloud, all the unknown unknowns.

Re: Don't rent the cloud, own instead

#420

Post by toomuchtodo »

organsnyder wrote: post #401
toomuchtodo wrote: post #269

Earlier quoted context omitted.

Perhaps we need bootcamps for on prem stacks if we are concerned about a skills gap. This is no different imho from the trades skills shortage many developed countries face. The muscle must be flexed. Otherwise, you will be held captive by a provider "who does it all for you". "Today, we are going to calculate the power requirements for this rack, rack the equipment, wire power and network up, and learn how to use PX…

This might be my own ego talking (I see myself as a generalist), but IMHO what we need are people that are comfortable jumping into unfamiliar systems and learning on-the-fly, applying their existing knowledge to new domains (while recognizing the assumptions their existing knowledge is causing them to make). That seems much harder to teach, especially in a boot camp format.

As a very curious autodidact, I strongly agree, but this talent is rare and can punch it's own ticket (broadly speaking). These people innovate and build systems for others to maintain, in my experience. But, to your point, we should figure out the sorting hat for folks who want to radically own these on prem systems [1] if they are needed.

[1] https://xkcd.com/705/

Post reply on HN