Why we use our own hardware
fastmail.com
Why we use our own hardware
1–10 of 547 posts
Re: Why we use our own hardware
#2longtime FM user here
good on them, understanding infrastructure and cost/benefit is essential in any business you hope to run for the long haul
Re: Why we use our own hardware
#3Such an awesome article. I like how they didn't just go with the Cloud wave but kept sysadmin'ing, like ol' Unix graybeards. Two interesting things they wrote about their SSDs:
1) "At this rate, we’ll replace these [SSD] drives due to increased drive sizes, or entirely new physical drive formats (such E3.S which appears to finally be gaining traction) long before they get close to their rated write capacity."
and
2) "We’ve also anecdotally found SSDs just to be much more reliable compared to HDDs (..) easily less than one tenth the failure rate we used to have with HDDs."
Re: Why we use our own hardware
#4I like this writeup, informative and to-the-point.
Today, the cloud isn’t about other people’s hardware.
It’s about infrastructure being an API call away. Not just virtual machines but also databases, load-balancers, storage, and so on.
The cost isn’t the DC or the hardware, but the hours spend on operations.
And you can abuse developers to do operations on the side :-)
Re: Why we use our own hardware
#5Didn’t see this in the article, do they have multi az redundancy? I.e. if the entire raid goes up in flames what’s the recovery process?
Re: Why we use our own hardware
#6Would be interesting to know how files get stored. They don't mention any distributed FS solutions like SeaweedFS so once a drive is full, does the file get sent to another one via some service? Also ZFS seems an odd choice since deletions (esp of small files) at +80% full drive are crazy slow.
Re: Why we use our own hardware
#7I like this writeup, informative and to-the-point. Today, the cloud isn’t about other people’s hardware. It’s about infrastructure being an API call away. Not just virtual machines but also databases, load-balancers, storage, and so on. The cost isn’t the DC or the hardware, but the hours spend on operations. And you can abuse developers to do operations on the side :-)
And then come the weird aspects of bad cloud service providers, like IONOS, who have broken OS images, a provisioning API, that is a bottleneck, where what other people do and how much they do can slow down your own provisioning and creating network interfaces can take minutes via their API and their customer services says "That's how it is, cannot change it.", and you get a very shitty web user interface, that desperately tries to be a single page app, yet has all the default browser functionality like the back button broken. Yet they still cost literally 10x what Hetzner cloud costs, while Hetzner basically does everything better.
And then it is still also about other people's hardware in addition to that.
Re: Why we use our own hardware
#8Such an awesome article. I like how they didn't just go with the Cloud wave but kept sysadmin'ing, like ol' Unix graybeards. Two interesting things they wrote about their SSDs: 1) "At this rate, we’ll replace these [SSD] drives due to increased drive sizes, or entirely new physical drive formats (such E3.S which appears to finally be gaining traction) long before they get close to their rated write capacity." and 2)…
To avoid sysadmin tasks, and keep costs down, you've got to go so deep in the cloud, that it becomes just another arcane skill set. I run most of my stuff on virtual Linux servers, but some on AWS, and that's hard to learn, and doesn't transfer to GCP or Azure. Unless your needs are extreme, I think sysadmin'ing is the easier route in most cases.
Re: Why we use our own hardware
#9Would be interesting to know how files get stored. They don't mention any distributed FS solutions like SeaweedFS so once a drive is full, does the file get sent to another one via some service? Also ZFS seems an odd choice since deletions (esp of small files) at +80% full drive are crazy slow.
The open-source Cyrus IMAP server which they mention using, has replication built-in. ZFS also has built-in replication available.
Deletion of files depends on how they have configured the message store - they may be storing a lot of data into a database, for example.
Re: Why we use our own hardware
#10Such an awesome article. I like how they didn't just go with the Cloud wave but kept sysadmin'ing, like ol' Unix graybeards. Two interesting things they wrote about their SSDs: 1) "At this rate, we’ll replace these [SSD] drives due to increased drive sizes, or entirely new physical drive formats (such E3.S which appears to finally be gaining traction) long before they get close to their rated write capacity." and 2)…
The power of Moore's law.