Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

371–380 of 516 posts

Re: Don't rent the cloud, own instead

#371

Post by walt_grata »

infecto wrote: post #194
bojangleslover wrote: post #179

Earlier quoted context omitted.

The complexity is what gets you. One of AWS's favorite situations is 1) Senior engineer starts on AWS 2) Senior engineer leaves because our industry does not value longevity or loyalty at all whatsoever (not saying it should, just observing that it doesn't) 3) New engineer comes in and panics 4) Ends up using a "managed service" to relieve the panic 5) New engineer leaves 6) Second new engineer comes in and not only…

It’s all anecdotal but in my experiences it’s usually opposite. Bored senior engineer wants to use something new and picks a AWS bespoke service for a new project. I am sure it happens a multitude of ways but I have never seen the case you are describing.

Is that what I should be doing? I'm just encouraging the devs on my team to read designing data intensive apps and setting up time for group discussions. Aside from coding and meetings that is.

Re: Don't rent the cloud, own instead

#373

Post by phil21 »

friendzis wrote: post #235
couscouspie wrote: post #221

Earlier quoted context omitted.

That's not a factual statement over reality, but more of a normative judgement to justify resignation. Yes, professionals that know how to actually do these things are not abundantly available, but available enough to achieve the transition. The talent exists and is absolutely passionate about software freedom and hence highly intrinsically motivated to work on it. The only thing that is lacking so far is the demand…

> and the talent available will skyrocket, when the market starts demanding it. Part of what clouds are selling is experience. A "cloud admin" bootcamp graduate can be a useful "cloud engineer", but it takes some serious years of experience to become a talented on prem sre. So it becomes an ouroboros: moving towards clouds makes it easier to move to the clouds.

> A "cloud admin" bootcamp graduate can be a useful "cloud engineer",

If by useful you mean "useful at generating revenue for AWS or GCP" then sure, I agree.

These certificates and bootcamps are roughly equivalent to the Cisco CCNA certificate and training courses back in the 90's. That certificate existed to sell more Cisco gear - and Cisco outright admitted this at the time.

Re: Don't rent the cloud, own instead

#374

Post by Torq_boi »

pu_pe wrote: post #58

> Self-reliance is great, but there are other benefits to running your own compute. It inspires good engineering. It's easy to inspire people when you have great engineers in the first place. That's a given at a place like comma.ai, but there are many companies out there where administering a datacenter is far beyond their core competencies. I feel like skilled engineers have a hard time understanding the trade-offs…

We actually do have an in-house chef lol.

Re: Don't rent the cloud, own instead

#375

Post by vadepaysa »

I was an on-prem maxi (if thats a thing) for a long time. I've run clusters that costed more than $5M, but these days I am a changed man. I start with PaaS like Vercel and work my way down to on-prem depending on how important and cost conscious that workload is.

Pains I faced running BIG clusters on-prem.

1. Supply chain Management -- everything from power supplies all the way to GPUs and storage has to be procured, shipped, disassembled and installed. You need labor pool and dedicated management.

2. Inventory Management -- You also need to manage inventory on hand for parts that WILL fail. You can expect 20% of your cluster to have some degree of issues on an ongoing basis

3. Networking and security -- You are on your own defending your network or have to pay a ton of money to vendors to come in and help you. Even with the simplest of storage clusters, we've had to deal with pretty sophisticated attacks.

When I ran massive clusters, I had a large team dealing with these. Obviously, with PaaS, you dont need anyone.

Re: Don't rent the cloud, own instead

#376

Post by unethical_ban »

infecto wrote: post #366
unethical_ban wrote: post #343

Earlier quoted context omitted.

SV and financial services are quite different. It's 2026 and banks are still running their mainframe, running windows VMs on VMware and building their enterprise software with Java. The big boys still have their own datacenters they own. Sure, they try dabbling with cloud services, and maybe they've pushed their edge out there, and some minor services they can afford to experiment with.

If you are working at a bank you are most likely not standing up your own Postgres and related services. Even 15 years ago. I am not saying it never happened, I am saying that even 15 years ago even large orgs with data enters often had in place sys and devops that helped with providing resources. Obviously not the rule but also not an exception.

True. We had separate teams for Oracle and MSSQL management. We had 3 teams each for Windows, "midrange" (Unix) and mainframe server management. That doesn't include IAM.

Re: Don't rent the cloud, own instead

#377

Post by weavie »

geocar wrote: post #123
weavie wrote: post #110

Earlier quoted context omitted.

What is the upper limit of Hertzner? Say you have an AWS bill in the $100s of millions, could Hertzner realistically take on that scale?

Who are you thinking of? Netflix might be spending as much as $120m (but probably a little less), and I thought they were probably Amazon's biggest customer. Does someone (single-buyer) spend more than that with AWS? Hertzner's revenue is somewhere around $400m, so probably a little scary taking on an additional 30% revenue from a single customer, and Netflix's shareholders would probably be worried about risk relyin…

I'm largely just thinking $HUGE when throwing out that number, but there are plenty of companies that have cloud costs in that range. A quick search brings up Walmart, Meta, Netflix, Spotify, Snap, JP Morgan.

Re: Don't rent the cloud, own instead

#378

Post by Torq_boi »

speedgoose wrote: post #49

I would suggest to use both on-premise hardware and cloud computing. Which is probably what comma is doing. For critical infrastructure, I would rather pay a competent cloud provider than being responsible for reliability issues. Maintaining one server room in the headquarters is something, but two servers rooms in different locations, with resilient power and network is a bit too much effort IMHO. For running many s…

Yes, we still use the azure for user-facing services and the website. They don't need GPUs and don't need expensive resources, so it's not as worth it to bring those in-house.

We also rely on github. It has historically been good a service, but getting worth it.

Re: Don't rent the cloud, own instead

#380

Post by majormajor »

vadepaysa wrote: post #375

I was an on-prem maxi (if thats a thing) for a long time. I've run clusters that costed more than $5M, but these days I am a changed man. I start with PaaS like Vercel and work my way down to on-prem depending on how important and cost conscious that workload is. Pains I faced running BIG clusters on-prem. 1. Supply chain Management -- everything from power supplies all the way to GPUs and storage has to be procured,…

In addition to those sorts of non-first-hardware-purchase costs, the person writing the check needs to think long and hard about how bad an outage would be, and how much money it makes sense to budget simply to "avoiding outages." And the more important it is not to have any downtime, the more it's gonna cost to build up some sort of substitute for cross-datacenter cloud functionality. (You are also likely not going to be as good at either managing and configuring those networks, or hiring people to do so, as AWS, either.)
Post reply on HN