Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use our own hardware

fastmail.com

Post reply on HN
Print view View original

441–450 of 547 posts

Re: Why we use our own hardware

#441

Post by evantbyrne »

Xylakant wrote: post #359
evantbyrne wrote: post #348

Earlier quoted context omitted.

Painful is an understatement. Keys for sensitive customer data should be derived from customer secrets either way. Almost nobody does that though, because it requires actual forethought. Instead they just slap secrets in KMS and pretend it's better than encrypted environment variables or other secrets services. If an attacker can read your secrets with the same level of penetration into your system, then it's all the…

There are many kinds of secrets that are used for purposes where they cannot be derived from customer secrets, and those still need to be secured. TLS private keys for example. I do disagree on the second part - there’s a world of a difference whether an attacker obtains a copy of your certificates private key and can impersonate you quietly or whether they gain the capability to perform signing operations on your be…

It's all unencrypted secrets from perspective of an attacker. If they somehow already have enough access to read your environment variables, then they can definitely access secrets manager records authorized for that service. By all means put secrets management in a secondary service to prevent leaking keys, but you don't need a cloud service to do that.

Re: Why we use our own hardware

#442

Post by leshenka »

bluGill wrote: post #404
llm_trw wrote: post #373

Earlier quoted context omitted.

>As the fortunes of AWS et al rose and rose and rose, I kept looking at their pricing at features and kept wondering what I was missing. They seemed orders of magnitude more expensive for something that was more complex to manage and would have locked us into a specific vendor's tooling. But everyone seemed to be flocking to them. In 2006 when the first aws instances showed up it would take you two years of on demand…

Aws is useful if you have uneven loads. why pay for the number of servers you need for christmas the rest of the year? But if your load is more even it doesn't make as much sense.

But how does it look from aws point of view?

Everyone scales up around Christmas then scales down afterwards. What do THEY do with all the unneeded CPU-seconds for the rest of the year?

Re: Why we use our own hardware

#443

Post by wutwutwat »

dijit wrote: post #397
packtreefly wrote: post #356

Earlier quoted context omitted.

> although I was worried that folks are too locked in to SaaS stuff For some people the cloud is straight magic, but for many of us, it just represents work we don't have to do. Let "the cloud" manage the hardware and you can deliver a SaaS product with all the nines you could ask for... > teaching a course on how to do all this ... there might be interest in that after all? Idk about a course, but I'd be interested…

I’ve never visited a datacenter that wasn’t SOC2 compliant. Bahnhof, SAVVIS, Telecity, Equinox etc. Of course, their SOC 2 compliance doesn't mean we are absolved of securing our databases and services. Theres a big gap between throwing some compute in a closet and having someone “run the closet” for you. There is, a significantly larger gap between having someone “run the closet” and building your own datacenter fro…

A datacenter being soc2 compliant doesn’t mean any of your systems are. Same with pci. Same with hipaa. Cloud providers usually have offerings that help meet those requirements as well, but again, you can host bare metal, colo, cloud, or a tower under your bed, their compliance doesn’t do anything to cover your compliance.

Re: Why we use our own hardware

#444

Post by champtar »

sgarland wrote: post #116
the__alchemist wrote: post #110

Earlier quoted context omitted.

Do you need those tools? It seems that for fundamental web hosting, you need your application server, nginx or similar, postgres or similar, and a CLI. (And an interpreter etc if your application is in an interpreted lang)

I suppose that depends on your RTO. With cloud providers, even on a bare VM, you can to some extent get away with having no IaC, since your data (and therefore config) is almost certainly on networked storage which is redundant by design. If an EC2 fails, or even if one of the drives in your EBS drive fails, it'll probably come back up as it was. If it's your own hardware, if you don't have IaC of some kind – even so…

All EBS volumes except io2 have advertised durability of 99.8%, which is pretty low, so don't count it in the magic networked storage category.

Re: Why we use our own hardware

#445

Post by bluGill »

leshenka wrote: post #442
bluGill wrote: post #404

Earlier quoted context omitted.

Aws is useful if you have uneven loads. why pay for the number of servers you need for christmas the rest of the year? But if your load is more even it doesn't make as much sense.

But how does it look from aws point of view? Everyone scales up around Christmas then scales down afterwards. What do THEY do with all the unneeded CPU-seconds for the rest of the year?

Not everyone. Ag is in a low time around then and scales way back. I don't know what other industry is like.

Re: Why we use our own hardware

#446

Post by 0xbadcafebee »

kapone wrote: post #438
0xbadcafebee wrote: post #375

Earlier quoted context omitted.

You know how to set up a rock-solid remote hands console to all your servers, I take it? Dial-up modem to a serial console server, serial cables to all the servers (or IPMI on a segregated network and management ports). Then you deal with varying hardware implementations, OSes, setting that up in all your racks in all your colos. Compare that to AWS, where there are 6 different kinds of remote hands, that work on all…

ok...? But, it comes at a cost. And that cost is significant. Like magnitudes significant. At what point does it become cheaper to hire an infra engineer? Let's see. In the US a good infra engineer might cost you $150K/yr all in. That's not taking into account freelancers/contractors who can do it for less. That's ~$12K/mo. That's a lot of compute on AWS...but that's not the end of the story. Ever try getting data OU…

The bigger cost is what will happen to your business when you're hard-down for a week because all your SQL servers are down, and you don't have spares, and it will take a week to ship new servers and get them racked. Even if you think you could do that very fast, there is no guarantee. I've seen Murphy's Law laugh in the face of assumptions and expectations too many times.

But let's not just make vague claims. Everybody keeps saying AWS is more expensive, right? So let's look at one random example: the cost of a server in AWS vs buying your own server in a colo.

  AWS:
    1x c6g.8xlarge (32-vCPU, 64GB RAM, us-east-2, Reserved Instance plan @ 3yrs)
       Cost up front: $5,719
       Cost over 3 years: $11,437 ($158.85/month + $5,719 upfront)

  On-prem:
    1x Supermicro 1U WIO A+ Server (AS -1115SV-WTNRT), 1x AMD EPYC™ 8324P Processor 32-Core 2.65GHz 128MB Cache (180W), 2x 32GB DDR5 5600MHz ECC RDIMM Server Memory, 2x 240GB 2.5" PM893 SATA 6Gb/s Solid State Drive (1 x DWPD), 3 Years Parts and Labor + 2 Years of Cross Shipment, MCP-290-00063-0N - Supermicro 1U Rail Kit (Included), 2 10GbE RJ45 Ports : $4,953.40
    1x Colo shared rack 1U 2-PS @ 120VAC: $120/month (100Mbps only)
      Cost up front: $4,953.40 (before shipping & tax)
      Cost over 3 years: $9,273 (minimum)
So, yes, the AWS server is double the cost (not an order of magnitude) of a ServerMicro (& this varies depending on configuration). But with colocation fees, remote hands fees, faster internet speeds, taxes, shipping, and all the rest of the nickle-and-diming, the cost of a single server in a colo is almost the same as AWS. Switch to a full rack, buy the networking gear, remote hands gear, APCs, etc that you'll probably want, and it's way, way more expensive to colo. In this one example.

Obviously, it all depends on a huge number of factors. Which is why it's better not to just take the copious number of "we do on-prem and everything is easy and cheap" stories at face value. Instead one should do a TCO analysis based on business risk, computing requirements, and the non-monetary costs of running your own micro-datacenter.

Re: Why we use our own hardware

#447

Post by AndrewDavis »

brongondwana wrote: post #410
bartvk wrote: post #3

Such an awesome article. I like how they didn't just go with the Cloud wave but kept sysadmin'ing, like ol' Unix graybeards. Two interesting things they wrote about their SSDs: 1) "At this rate, we’ll replace these [SSD] drives due to increased drive sizes, or entirely new physical drive formats (such E3.S which appears to finally be gaining traction) long before they get close to their rated write capacity." and 2)…

My beard isn't entirely grey yet! The new NVMe drives we've only had for a few years, but so far there's only been a single failure across the whole fleet, and we keep spares in stock. It's been very reliable, not like the weeks back in (hmm, 2006? 2007?) the ancient past, when we were losing 15kRPM velociraptors every other day. They had a firmware fault and we eventually got an update which made them reliable, but…

A few more than one, but it has been a lot less than when we were dealing with spinner. I think I requested about one or two replacements a year, a far cry from the one a week I was doing before.

Re: Why we use our own hardware

#448

Post by jedberg »

leshenka wrote: post #442
bluGill wrote: post #404

Earlier quoted context omitted.

Aws is useful if you have uneven loads. why pay for the number of servers you need for christmas the rest of the year? But if your load is more even it doesn't make as much sense.

But how does it look from aws point of view? Everyone scales up around Christmas then scales down afterwards. What do THEY do with all the unneeded CPU-seconds for the rest of the year?

Only consumer scales up for the holidays. Most other industries scale down. The more companies they have, the more even the overall demand is for them.

Also, every unused resource goes into the spot market. They just have a bigger spot market during the year.

And lastly, that's why they charge a premium. Because they amortize the cost of spare hardware across all their customers.

Re: Why we use our own hardware

#449

Post by brongondwana »

AndrewDavis wrote: post #447
brongondwana wrote: post #410

Earlier quoted context omitted.

My beard isn't entirely grey yet! The new NVMe drives we've only had for a few years, but so far there's only been a single failure across the whole fleet, and we keep spares in stock. It's been very reliable, not like the weeks back in (hmm, 2006? 2007?) the ancient past, when we were losing 15kRPM velociraptors every other day. They had a firmware fault and we eventually got an update which made them reliable, but…

A few more than one, but it has been a lot less than when we were dealing with spinner. I think I requested about one or two replacements a year, a far cry from the one a week I was doing before.

If I didn't see it, it didn't happen...

OK, I stand corrected. We lose one or two NVMe drives per year :)

Re: Why we use our own hardware

#450

Post by brongondwana »

0xbadcafebee wrote: post #446
kapone wrote: post #438

Earlier quoted context omitted.

ok...? But, it comes at a cost. And that cost is significant. Like magnitudes significant. At what point does it become cheaper to hire an infra engineer? Let's see. In the US a good infra engineer might cost you $150K/yr all in. That's not taking into account freelancers/contractors who can do it for less. That's ~$12K/mo. That's a lot of compute on AWS...but that's not the end of the story. Ever try getting data OU…

The bigger cost is what will happen to your business when you're hard-down for a week because all your SQL servers are down, and you don't have spares, and it will take a week to ship new servers and get them racked. Even if you think you could do that very fast, there is no guarantee. I've seen Murphy's Law laugh in the face of assumptions and expectations too many times. But let's not just make vague claims. Everyb…

Yeah, if you don't have levels of redundancy, then you're pretty screwed. We could theoretically lose 2/3 of our systems and have sufficient capacity, because our metric is 2N primary plus N secondary, and we can run with half the racks switched off in the primary, or with the secondary entirely switched off, or (in theory, there's still some kinks with failover) with just secondary.
Post reply on HN