Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use our own hardware

fastmail.com

Post reply on HN
Print view View original

351–360 of 547 posts

Re: Why we use our own hardware

#351

Post by lukevp »

To me, Cloud is all about the shift left of DevOps. It’s not a cost play. I’m a Dev Lead / Manager and have worked in both types of environments over the last 10 years. It’s immeasurable the velocity difference as far as system provisioning between the two approaches. In the hardware space, it took months to years to provision new machines or upgrade OSes. In the cloud, it’s a new terraform script and a CI deploy away. Need more storage? It’s just there, available all the time. Need to add a new firewall between machines or redo the network topology? Free. Need a warm standby in 4 different regions that costs almost nothing but can scale to full production capacity within a couple of minutes? Done. Those types of things are difficult to do with physical hardware. And if you have an engineering culture where the operational work and the development work are at odds (think the old style of Dev / QA / Networking / Servers / Security all being separate teams), processes and handoffs eat your lunch and it becomes crippling to your ability to innovate. Cloud and DevOps are to me about reducing the differentiation between these roles so that a single engineer can do any part of the stack, which cuts out the communication overhead and the handoff time and the processes significantly.

If you have predictable workloads, a competent engineering culture that fights against process culture, and are willing to spend the money to have good hardware and the people to man it 24x7x365 then I don’t think cloud makes sense at all. Seems like that’s what y’all have and you should keep up with it.

Re: Why we use our own hardware

#352

Post by swiftcoder »

shortsunblack wrote: post #245
swiftcoder wrote: post #198

Earlier quoted context omitted.

> All the pro-cloud talking points... don't persuade anyone with any real technical understanding This is a very engineer-centric take. The cloud has some big advantages that are entirely non-technical: - You don't need to pay for hardware upfront. This is critical for many early-stage startups, who have no real ability to predict CapEx until they find product/market fit. - You have someone else to point the SOC2/HIP…

You can over-provision your own baremetal resources 20x and it will be still cheaper than cloud. The capex talking point is just that, a talking point.

As an early-stage startup?

Your spend in the first year on AWS is going to be very close to zero for something like a SaaS shop.

Nor can you possibly scale in-house baremetal fast enough if you hit the fabled hockey stick growth. By the time you sign a colocation contract and order hardware, your day in the sun may be over.

Re: Why we use our own hardware

#353

Post by ksec »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

>But everyone seemed to be flocking to them.

To the point we have young Devs today that dont know what VPS and Colo ( Colocation) meant.

Back to the article, I am surprised it was only a "A few years ago" Fastmail adopted SSD. Which certainly seems late in the cycle for the benefits of what SSD offers.

Price for Colo on the order of $3000/2U/year. That is $125 /U/month.

Re: Why we use our own hardware

#355

Post by milesvp »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

As someone who lived through that era, I can tell you there are legions of devs and dev adjacent people who have no idea what it’s like to automate mission critical hardware. Everyone had to do it in the early 2000s. But it’s been long enough that there are people in the workforce who just have no idea about running your own hardware since they never had to. I suspect there is a lot of interest, especially since we’re likely approaching the bring it back in house cycle, as CTOs try to reign in their cloud spend.

Re: Why we use our own hardware

#356

Post by packtreefly »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

> although I was worried that folks are too locked in to SaaS stuff

For some people the cloud is straight magic, but for many of us, it just represents work we don't have to do. Let "the cloud" manage the hardware and you can deliver a SaaS product with all the nines you could ask for...

> teaching a course on how to do all this ... there might be interest in that after all?

Idk about a course, but I'd be interested in a blog post or something that addresses the pain points that I conveniently outsource to AWS. We have to maintain SOC 2 compliance, and there's a good chunk of stuff in those compliance requirements around physical security and datacenter hygiene that I get to just point at AWS for.

I've run physical servers for production resources in the past, but they weren't exactly locked up in Fort Knox.

I would find some in-depth details on these aspects interesting, but from a less-clinical viewpoint than the ones presented in the cloud vendors' SOC reports.

Re: Why we use our own hardware

#357

Post by MadnessASAP »

jameshart wrote: post #281
j45 wrote: post #271

Earlier quoted context omitted.

Nope. It's literally putting private data in a shared drawer in someone else's desk where you have your area of the drawer.

Literally? I would just like to point out that most of us who have ever had a job at an office, attended an academic institution, or lived in rented accommodation have kept stuff in someone else’s desk drawer from time to time. Often a leased desk in a building rented from a random landlord. Keeping things in someone else’s desk drawer can be convenient and offer a sufficient level of privacy for many purposes. And y…

> Keeping things in someone else’s desk drawer can be convenient and offer a sufficient level of privacy for many purposes.

Too torture a metaphor to death, are you going to keep your bank passwords in somebody else's desk drawer? Are you going to keep 100 million people's bank passwords in that drawer?

> I guess, since I’m not a carpenter, I can buy a flatpack desk from ikea and assemble it and keep my stuff in that. I’m not sure that’s an improvement to my privacy posture in any meaningful sense though.

If you're not a carpenter I would recommend you stay out of the business of building safe desk drawers all together. Although you should probably still be able to recognize that the desk drawer you own, that is inside your own locked house is a safer option then the one at the office accessible by any number of people.

Re: Why we use our own hardware

#358

Post by benterix »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

> As the fortunes of AWS et al rose and rose and rose, I kept looking at their pricing at features and kept wondering what I was missing.

You are not the only one. There are several factors at play but I believe one of the strongest today is the generational divide: the people lost the ability to manage their own infra or don't know it well enough to do it well so it's true when they say "It's too much hassle". I say this as an AWS guy who occasionally works on on-prem infra.[0]

[0] As a side note, I don't believe the lack of skills is the main reason organizations have problem - skills can be learned, but if you mess up the initial architecture design, fixing that can easily take years.

Re: Why we use our own hardware

#359

Post by Xylakant »

evantbyrne wrote: post #348
Xylakant wrote: post #346

Earlier quoted context omitted.

An attacker gaining temporary capability to encrypt/decrypt data through a compromised instance is painful. An attacker gaining a copy of a private key is still an entirely different world of pain.

Painful is an understatement. Keys for sensitive customer data should be derived from customer secrets either way. Almost nobody does that though, because it requires actual forethought. Instead they just slap secrets in KMS and pretend it's better than encrypted environment variables or other secrets services. If an attacker can read your secrets with the same level of penetration into your system, then it's all the…

There are many kinds of secrets that are used for purposes where they cannot be derived from customer secrets, and those still need to be secured. TLS private keys for example.

I do disagree on the second part - there’s a world of a difference whether an attacker obtains a copy of your certificates private key and can impersonate you quietly or whether they gain the capability to perform signing operations on your behalf temporarily while they maintain access to a compromised instance.

Re: Why we use our own hardware

#360

Post by wslh »

portaouflop wrote: post #349
wslh wrote: post #316

Earlier quoted context omitted.

You can have full security competence but be a rogue actor at the same time.

You can also have rogue actors in your company, you don’t need 3rd parties for that

That doesn't sum up my comments in the thread. A rogue actor in a datacenter could attack zillions of companies at the same time while rogue actors in a single company only once.
Post reply on HN