Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use our own hardware

fastmail.com

Post reply on HN
Print view View original

411–420 of 547 posts

Re: Why we use our own hardware

#411

Post by brongondwana »

DarkCrusader2 wrote: post #84

I have seen a common sentiment that self hosting is almost always better than cloud. What these discussions does not mention is how to effectively run your business applications on this infrastructure. Things like identity management (AAD/IAM), provisioning and running VMs, deployments. Network side of things like VNet, DNS, securely opening ports etc. Monitoring setup across the stack. There is so much functionaliti…

We're (very boring I know) just putting it all in a git repository with a Makefile which deploys it, plus some basic orchestration to run 'make diff' across the cluster and see what's out of sync, and 'make install' across hosts to deploy it into pl ace.

It's clunky, but simple, repeatable, and easily (vsfo) understood.

As for the bigger things, software etc - we have scripts that generate Debian packages which we store in our own private repo. You just install `fastmail-server` and the dependency management updates everything. There's a daily cronjob which checks if there are updated security packages or thing we failed to correctly deploy and emails us as well.

It's amazing what you can build on top of the OS provided tools with not too much complexity if you don't overthink it.

Re: Why we use our own hardware

#412

Post by brongondwana »

jmakov wrote: post #6

Would be interesting to know how files get stored. They don't mention any distributed FS solutions like SeaweedFS so once a drive is full, does the file get sent to another one via some service? Also ZFS seems an odd choice since deletions (esp of small files) at +80% full drive are crazy slow.

Emails are stored in cyrus-imapd.

For now, the "file storage" product is a Node tree in mysql, with content stored in a content-addressed blob store, which is some custom crap I wrote 15 years ago that is still going strong because it's so simple there's not much to go wrong.

We do plan to eventually move the blob storage into Cyrus as well though, because then we have a single replication and backup system rather than needing separate logic to maintain the blob store.

Re: Why we use our own hardware

#413

Post by brongondwana »

ryao wrote: post #46
jmakov wrote: post #6

Would be interesting to know how files get stored. They don't mention any distributed FS solutions like SeaweedFS so once a drive is full, does the file get sent to another one via some service? Also ZFS seems an odd choice since deletions (esp of small files) at +80% full drive are crazy slow.

Unlike ext4 that locks the directory when unlinking, ZFS is able to scale on parallel unlinking. In specific, ZFS has range locks that permit directory entries to be removed in parallel from the extendible hash trees that store them. While this is relatively slow for sequential workloads, it is fast on parallel workloads. If you want to delete a large directory subtree fast on ZFS, do the rm operations in parallel. F…

Unlinking gets done asynchronously on the weekends from Cyrus, using the `cyr_expire` tool. Right now it only runs one unlinking process at a time on the whole machine due to historical ext4 issues ... but maybe we should revisit that now we're on ZFS and NVMe. Thanks for the reminder.

Re: Why we use our own hardware

#414

Post by brongondwana »

mastax wrote: post #53
shrubble wrote: post #9

Earlier quoted context omitted.

The open-source Cyrus IMAP server which they mention using, has replication built-in. ZFS also has built-in replication available. Deletion of files depends on how they have configured the message store - they may be storing a lot of data into a database, for example.

ZFS replication is quite unreliable when used with ZFS native encryption, in my experience. Didn't lose data but constant bugs.

Yeah, we're only using ZFS replication for logs; we're using the Cyrus replication for emails because it has other sanity checks and data model consistency enforcement which is really valuable.

(And both are async. We'd need something like drbd for real synchronous replication)

Re: Why we use our own hardware

#415

Post by brongondwana »

Amfy wrote: post #377
goldeneye13_ wrote: post #5

Didn’t see this in the article, do they have multi az redundancy? I.e. if the entire raid goes up in flames what’s the recovery process?

I believe they replicate from NJ to WA (Seattle). At least that's something they spoke about many years ago.

PHL to STL these days, but same design:

https://www.fastmail.com/blog/moving-house-new-datacentre/

Re: Why we use our own hardware

#416

Post by brongondwana »

mlfreeman wrote: post #52
caidan wrote: post #14

I absolutely love Fastmail. I moved off of Gmail years ago with zero regrets. Better UI, better apps, better company, and need I say better service? I still maintain and fetch from a Gmail account so it all just works seamlessly for receiving and sending Gmail, so you don’t have to give anything up either.

I moved from my own colocated 1U running Mailcow to Fastmail and don't regret it one bit. This was an interesting read, glad to see they think things through nice and carefully. The only things I wish FM had are all software: 1. A takeout-style API to let me grab a complete snapshot once a week with one call 2. The ability to be an IdP for Tailscale.

1. hoping to have a JMAP archive format at some point which should cover that. I'd hope that normally you'd be fetching a delta update rather than the whole thing. We've got enough bandwidth for a few people do to it, but I wouldn't want every customer pulling their entire archive every week of 99% the same immutable data; that would be kinda sucky.

2. yeah, I'd love that too - we're keen to integrate with everything else that people are using. We have a basic in-house IdP thing for our own staff to authenticate against our hosted services, but haven't scaled it out. This will happen eventually, though I've been burned enough times I don't want to promise a timeframe.

Re: Why we use our own hardware

#417

Post by brongondwana »

herf wrote: post #279

zfs encryption is still corrupting datasets when using zfs send/receive for backup (huge win for mail datasets), would be cautious about using it in production: https://github.com/openzfs/zfs/issues/12014

yeah, we use Cyrus replication still - it's protocol specific so it detects changes very efficiently as well, using the internal MODSEQ system also used for the JMAP /changes and IMAP CONDSTORE/QRESYNC.

Plus it has protocol consistency sanity checks built in.

Plus, I wrote it :p

Re: Why we use our own hardware

#418

Post by sneak »

_bare_metal wrote: post #57

Plugging https://BareMetalSavings.com in case you want to ballpark-estimate your move off of the cloud Bonus points: I'm a Fastmail customer, so it tangentially tracks ---- Quick note about the article: ZFS encryption can be flaky, be sure you know what you're doing before deploying for your infrastructure. Relevant Reddit discussion: https://www.reddit.com/r/zfs/comments/1f59zp6/is_zfs_encrypt... A spreadsheet of re…

My main issue with ZFS encryption is that it only supports one key.

LUKS2 has something like 9 key slots.

I run ZoL over LUKS2 and it works great.

Re: Why we use our own hardware

#419

Post by brongondwana »

0xbadcafebee wrote: post #278

I've been doing this job for almost as long as they have. I work with companies that do on-prem, and I work with companies in the cloud, and both. Here's the low down: 1. The cost of the server is not the cost of on-prem. There are so many different kinds of costs that aren't just monetary. ("we have to do more ourselves, including planning, choosing, buying, installing, etc, ") Those are tasks that require expertise…

Yeah, we have good vendor relationships, good datacenter relationships, and we've made mis-steps along the way for sure. Own hardware isn't for everyone, but it's been great for us. YMMV
Post reply on HN