Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

481–490 of 516 posts

Re: Don't rent the cloud, own instead

#481

Post by geocar »

varsketiz wrote: post #424
geocar wrote: post #323

Earlier quoted context omitted.

> A $120M spend on AWS is equivalent to around a $12M spend on Hetzner Dedicated (likely even less, the factor is 10-20x in my experience), so that would be 3% of their revenue from a single customer. I'm not convinced. I assume someone at Netflix has thought about this, because if that were true and as simple as you say, Netflix would simply just buy Hetzner. I think there lots of reasons you could have this experie…

We run our own infrastructure, sometimes with our own fincing (4), sometimes external (3). The cost is in tens of millions per year. When I used to compare to aws, only egress at list price costs as much as my whole infra hosting. All of it. I would be very interested to understand why netflix does not go 3/4 route. I would speculate that they get more return from putting money in optimising costs for creating origin…

> I would be very interested to understand why netflix does not go 3/4 route. I would speculate that they get more return from putting money in optimising costs for creating original content, rather than cloud bill.

I invest in Netflix, which means I'm giving them some fast cash to grow that business.

I'm not giving them cash so that they can have cash.

If they share a business plan that involves them having cash to do X, I wonder why they aren't just taking my cash to do X.

They know this. That's why on the investors calls they don't talk about "optimising costs" unless they're in trouble.

I understand self-hosting and self-building saves money in the long-long term, and so I do this in my own business, but I'm also not a public company constantly raising money.

> When I used to compare to aws, only egress at list price costs as much as my whole infra hosting. All of it.

I'm a mere 0,1% of your spend, and I get discounts.

You would not be paying "list price".

Netflix definitely would not be.

Re: Don't rent the cloud, own instead

#482

Post by conductr »

tedd4u wrote: post #476
conductr wrote: post #465

Earlier quoted context omitted.

Believe you're talking about conserving cash through reduced taxes since this guy was against paying taxes. However, spending a premium on cloud services over what you could with an on-prem capital investment does not help your cash position. His tenant of frugality would have conflicted, especially since the cloud premium can easily exceed the tax rate - that is to say, paying taxes would have been cheaper Section i…

Probably depends on where your gross margins would be with cloud and if you're higher or lower growth. If cloud will let you grow faster (HA/DR on-prem is hard) and you'll still have 75-80%+ gross margins, why slow top-line growth to do on-prem?

It’s not a real concern for vast majority of businesses. It’s a common excuse but practically no business is outgrowing a cheaper than cloud solution. Maybe on-prem isn’t right first step, but that doesn’t force you to cloud. There’s dedicated servers and everything in between.

On prem is maybe not the best first step but Colo or dedicated servers gives you a cleaner path to going on-prem if you ever decide to. The cost of growth is too high in cloud.

Learning how to run servers is actually less complicated than all the cloud architecture stuff and doesn’t have to be slower. There’s no one sized fits all, but I believe old boring solutions should be employed first and could be used to run most applications. Technology has a way of getting more complex every year just to accomplish the same tasks. But that’s largely optional.

Re: Don't rent the cloud, own instead

#483

Post by varsketiz »

geocar wrote: post #481
varsketiz wrote: post #424

Earlier quoted context omitted.

We run our own infrastructure, sometimes with our own fincing (4), sometimes external (3). The cost is in tens of millions per year. When I used to compare to aws, only egress at list price costs as much as my whole infra hosting. All of it. I would be very interested to understand why netflix does not go 3/4 route. I would speculate that they get more return from putting money in optimising costs for creating origin…

> I would be very interested to understand why netflix does not go 3/4 route. I would speculate that they get more return from putting money in optimising costs for creating original content, rather than cloud bill. I invest in Netflix, which means I'm giving them some fast cash to grow that business. I'm not giving them cash so that they can have cash. If they share a business plan that involves them having cash to…

Of course netflix is optimising costs, otherwise it would not be a business, I just think they put much more effort elsewhere. They could be using other words, like "financial discipline" :)

My point is that even if I get 20 times discount on egress its still nowhere close, since i have to buy everything else - compute, storage are more expensive, and even with 5-10x discounts from list price its not worth it.

(Our cloud bills are in the millions as well, I am familiar with what discounts we can get)

Re: Don't rent the cloud, own instead

#484

Post by Symbiote »

sanderjd wrote: post #448
Dylan16807 wrote: post #412

Earlier quoted context omitted.

> Same here. But, I assume you have managed PostgreSQL in the past. I have. There are a large number of people software devs who have not. For them, it is not a low complexity task. And I can understand that. I'd say needing to read the documentation for the first time is what bumps it up from low complexity to medium. And then at medium you should still do it if there's a significant cost difference.

But is there a significant cost difference? I'm skeptical.

I hope this is the correct service, "Amazon RDS for PostgreSQL"? [1]

The main pair of PostgreSQL servers we have at work each have two 32-core (64-vthread) CPUs, so I think that's 128 vCPU each in AWS terms. They also have 768GiB RAM. This is more than we need, and you'll see why at the end, but I'll be generous and leave this as the default the calculator suggests, which is db.m5.12xlarge with 48 vCPU and 192GiB RAM.

That would cost $6559/month, or less if reserved which I assume makes sense in this case — $106400 for 3 years.

Each server has 2TB of RAID disk, of which currently 1TB is used for database data.

That is an additional $245/month.

"CloudWatch Database Insights" looks to be more detailed than the monitoring tool we have, so I will exclude that ($438/month) and exclude the auto-failover proxy as ours is a manual failover.

With the 3-year upfront cost this is $115000, or $192000 for 5 years.

Alternatively, buying two of yesterday's [2] list-price [3] Dell servers which I think are close enough is $40k with five years warranty (including next-business-day replacement parts as necessary).

That leaves $150000 for hosting, which as you can see from [4] won't come anywhere close.

We overprovision the DB server so it has the same CPU and RAM as our processing cluster nodes — that means we can swap things around in some emergency as we can easily handle one fewer cluster node, though this has never been necessary.

When the servers are out of warranty, depending on your business, you may be able to continue using them for a non-prod environment. Significant failures are still very unusual, but minor failures (HDDs) are more common and something we need to know how to handle anyway.

[1] https://calculator.aws/#/createCalculator/RDSPostgreSQL

[2] https://news.ycombinator.com/item?id=46899042

[3] There are significant discounts if you order regularly, buy multiple servers, or can time purchases when e.g. RAM is cheaper.

[4] https://www.voxility.com/colocation/prices

Re: Don't rent the cloud, own instead

#485

Post by Symbiote »

Dylan16807 wrote: post #412
cheema33 wrote: post #381

Earlier quoted context omitted.

> Managing the PostgreSQL databases is a medium to low complexity task as I see it. Same here. But, I assume you have managed PostgreSQL in the past. I have. There are a large number of people software devs who have not. For them, it is not a low complexity task. And I can understand that. I am a software dev for our small org and I run the servers and services we need. I use ansible and terraform to automate as much…

> Same here. But, I assume you have managed PostgreSQL in the past. I have. There are a large number of people software devs who have not. For them, it is not a low complexity task. And I can understand that. I'd say needing to read the documentation for the first time is what bumps it up from low complexity to medium. And then at medium you should still do it if there's a significant cost difference.

You can ask an AI or StackOverflow or whatever for the correct way to start a standby replica, though I think the PostgreSQL documentation is very good.

But if you were in my team I'd expect you to have read at least some of the documentation for any service you provision (self-hosted or cloud) and be able to explain how it is configured, and to document any caveats, surprises or special concerns and where our setup differs / will differ from the documented default. That could be comments in a provisioning file, or in the internal wiki.

That probably increases our baseline complexity since "I pressed a button on AWS YOLO" isn't accepted. I think it increases our reliability and reduces our overall complexity by avoiding a proliferation of services.

Re: Don't rent the cloud, own instead

#486

Post by Symbiote »

sanderjd wrote: post #407
objektif wrote: post #352

Earlier quoted context omitted.

You are right. Are you actually seriously considering whether to go fully managed or self managed at this point? Pls go AWS route and thank me later :)

No not at all, I have the same opinion as you! But I'm curious to understand the opposite view.

I ran through roughly our numbers here [1], it looks like self-hosted costs us about 25% of AWS.

I didn't include labour costs, but the self-hosted tasks (set up of hardware, OS, DB, backup, monitoring, replacing a failed component which would be really unusual) are small compared to the labour costs of the DB generally (optimizing indices, moving data around for testing etc, restoring from a backup).

[1] https://news.ycombinator.com/item?id=46910521

Re: Don't rent the cloud, own instead

#487

Post by Symbiote »

riku_iki wrote: post #398
Symbiote wrote: post #309

Earlier quoted context omitted.

Managing the PostgreSQL databases is a medium to low complexity task as I see it. Take two equivalent machines, set up with streaming replication exactly as described in the documentation, add Bacula for backups to an off-site location for point-in-time recovery. We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'…

> We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'd be scared of. this part is actually scariest, since there are like 10 different 3rd party solutions of unknown stability and maintanability.

I think if something like that is worrisome, I'd contact a PostgreSQL consultant for advice.

AWS charge about $500/month for this, so there's plenty of room to pay a consultant and still come out way ahead.

Re: Don't rent the cloud, own instead

#488

Post by maccard »

meatmanek wrote: post #328
maccard wrote: post #64

Earlier quoted context omitted.

We have two on site servers that we use. For various reasons (power cuts, internet outages, cleaners unplugging them) I’d say we have to intervene with them physically about once a month. It’s a total pain in the ass, especially when you don’t have _an_ it person sitting in the office to mind it. I’m in the Uk and our office is in Spain… But it is significantly cheaper and faster

You might want to look into colocating that server at a datacenter nearby. You can get a few U of rack space and the risk of power outages, internet outages, or cleaners unplugging the servers should go way down.

At that point it’s not worth it any more and we would just switch to hetzner or something instead.

Re: Don't rent the cloud, own instead

#489

Post by rcxdude »

antonvs wrote: post #315
bojangleslover wrote: post #179

Earlier quoted context omitted.

The complexity is what gets you. One of AWS's favorite situations is 1) Senior engineer starts on AWS 2) Senior engineer leaves because our industry does not value longevity or loyalty at all whatsoever (not saying it should, just observing that it doesn't) 3) New engineer comes in and panics 4) Ends up using a "managed service" to relieve the panic 5) New engineer leaves 6) Second new engineer comes in and not only…

> One of AWS's favorite situations I'll give you an alternative scenario, which IME is more realistic. I'm a software developer, and I've worked at several companies, big and small and in-between, with poor to abysmal IT/operations. I've introduced and/or advocated cloud at all of them. The idea that it's "more expensive" is nonsense in these situations. Calculate the cost of the IT/operations incompetence, and the c…

Fron what I've seen this can work as a stopgap until IT get their hooks into the cloud system in which case you circle back to paying to costs of incompetence and the costs of the cloud (sometimes stacking on top of each other).

Re: Don't rent the cloud, own instead

#490

Post by chrisandchris »

eru wrote: post #163
adamcharnock wrote: post #74

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50%…

> 4 - Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills, scale, cap-ex, and if you plan to run the servers for at least 3-5 years. Is it still the cheapest after you take into account that skills, scale, cap-ex and long term lock-in also have opportunity costs?

Personal experience: Did some cloud stuff for SME, and later on started colocation. I think my learning curve for all the cloud-stuff was the same as for all the colocation stuff, except the cloud will not get you rid of firewalls, NAT, DHCP and all that stuff. Cloud isn't that much easier, it's just a little bit different. IMHO, the largest disadvantage of colocation is that it requires (sometimes) physical presence at a datacenter.
Post reply on HN