A mail-cloud provider uses its own hardware? Well, that’s to be expected, it would be a refreshing article if it was written by one of their customers.
Why we use our own hardware
41–50 of 547 posts
Re: Why we use our own hardware
#42Didn’t see this in the article, do they have multi az redundancy? I.e. if the entire raid goes up in flames what’s the recovery process?
Looks like they do mention that elsewhere: https://www.fastmail.com/features/reliability/
> Fastmail has some of the best uptime in the business, plus a comprehensive multi data center backup system. It starts with real-time replication to geographically dispersed data centers, with additional daily backups and checksummed copies of everything. Redundant mirrors allow us to failover a server or even entire rack in the case of hardware failure, keeping your mail running.
Re: Why we use our own hardware
#43gmail does spam filtering very well for me. fastmail on the other hands, puts lots of legit emails into spam folder. manually marking "not spam" doesn't help
other than that, i'm happy with fastmail.
Re: Why we use our own hardware
#44But what about the cost and complexity of a room with the racks and the cooling needs of running these machines? And the uninterrupted power setup? The wiring mess behind the racks.
Even for cloud providers, these are mostly other people's problems, eg: Equinix
Re: Why we use our own hardware
#45Would be interesting to know how files get stored. They don't mention any distributed FS solutions like SeaweedFS so once a drive is full, does the file get sent to another one via some service? Also ZFS seems an odd choice since deletions (esp of small files) at +80% full drive are crazy slow.
Keeping enough free space should be much less of a problem with SSDs. They can tune it so the array needs to be 95% full before the slower best-fit allocator kicks in.
https://openzfs.readthedocs.io/en/latest/performance-tuning....
I think that 80% figure is from when drives were much smaller and finding free space over that threshold with the first-fit allocator was harder.
Re: Why we use our own hardware
#46Would be interesting to know how files get stored. They don't mention any distributed FS solutions like SeaweedFS so once a drive is full, does the file get sent to another one via some service? Also ZFS seems an odd choice since deletions (esp of small files) at +80% full drive are crazy slow.
Unlike ext4 that locks the directory when unlinking, ZFS is able to scale on parallel unlinking. In specific, ZFS has range locks that permit directory entries to be removed in parallel from the extendible hash trees that store them. While this is relatively slow for sequential workloads, it is fast on parallel workloads. If you want to delete a large directory subtree fast on ZFS, do the rm operations in parallel. For example, this will run faster on ZFS than a naive rm operation:
find /path/to/subtree -name -type f | parallel -j250 rm --
rm -r /path/to/subtree
For something like fast mail that has many users, unlinking should be parallel already, so unlinking on ZFS will not be slow for them.
By the way, that 80% figure has not been true for more than a decade. You are referring to the best fit allocator being used to minimize external fragmentation under low space conditions. The new figure is 96%. It is controlled by metaslab_df_free_pct in metaslab.c:
https://github.com/openzfs/zfs/blob/zfs-2.2.0/module/zfs/met...
Modification operations become slow when you are at/above 96% space filled, but that is to prevent even worse problems from happening. Note that my friend’s pool was below the 96% threshold when he was suffering from a slow rm -r. He just had a directory subtree with a large amount of directory entries he wanted to remove.
For what it is worth, I am the ryao listed here and I was around when the 80% to 96% change was made:
Re: Why we use our own hardware
#47But what about the cost and complexity of a room with the racks and the cooling needs of running these machines? And the uninterrupted power setup? The wiring mess behind the racks.
Do colocation facilities solve that?
Re: Why we use our own hardware
#48But what about the cost and complexity of a room with the racks and the cooling needs of running these machines? And the uninterrupted power setup? The wiring mess behind the racks.
There is a very competitive market for colo providers in basically every major metropolitan area in the US, Europe, and Asia. The racks, power, cooling, and network to your machines is generally very robust and clearly documented on how to connect. Deploying servers in house or in a colo is a well understood process with many experts who can help if you don’t have these skills.
Re: Why we use our own hardware
#49gmail does spam filtering very well for me. fastmail on the other hands, puts lots of legit emails into spam folder. manually marking "not spam" doesn't help other than that, i'm happy with fastmail.
iCloud is just as bad, sends important things to spam constantly and marking as “not spam” has never done anything perceivable.
Re: Why we use our own hardware
#50[flagged]
I get no spam on fastmail, I assume this is because I never give out my email to anyone and creating new ones for every interaction. This way I keep track of who I'm interacting with and also who's selling my alias emails. Just wish there was a decent way to do this with mobile numbers!
Same, I religiously create a masked email for every website (just checked, it's now at 163!). I simply don't give my "main" email out.
Oddly enough, simply unsubscribing from the things websites themselves has kept thing clean, I've yet to notice any true spam from a random source aimed at any of my emails since I joined last year.