Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use our own hardware

fastmail.com

Post reply on HN
Print view View original

371–380 of 547 posts

Re: Why we use our own hardware

#371

Post by flemhans »

ksec wrote: post #353
jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

>But everyone seemed to be flocking to them. To the point we have young Devs today that dont know what VPS and Colo ( Colocation) meant. Back to the article, I am surprised it was only a "A few years ago" Fastmail adopted SSD. Which certainly seems late in the cycle for the benefits of what SSD offers. Price for Colo on the order of $3000/2U/year. That is $125 /U/month.

HDDs are still the best option for many workloads, including email.

Re: Why we use our own hardware

#372

Post by brandon272 »

dehrmann wrote: post #229
UltraSane wrote: post #151

Earlier quoted context omitted.

"All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding," This is false. AWS infrastructure is vastly more secure than almost all company data centers. AWS has a rule that the same person cannot have logical access and physical access to the same storage device. Very few companies have enough IT people to have this rule. The AWS KMS is vastly m…

The other part is that when us-east-1 goes down, you can blame AWS, and a third of your customer's vendors will be doing the same. When you unplug the power to your colo rack while installing a new server, that's on you.

It's not always a full availability zone going down that is the problem. Also, despite the "no one ever got fired for buying Microsoft" logic, in practice I've never actually found stakeholders to be reassured by "its AWS and everyone is affected" when things are down. People want things back up and they want some informed answers about when that might happen, not "ehh its AWS, out of our control".

Re: Why we use our own hardware

#373

Post by llm_trw »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

>As the fortunes of AWS et al rose and rose and rose, I kept looking at their pricing at features and kept wondering what I was missing. They seemed orders of magnitude more expensive for something that was more complex to manage and would have locked us into a specific vendor's tooling. But everyone seemed to be flocking to them.

In 2006 when the first aws instances showed up it would take you two years of on demand bills to match the cost of buying the hardware from a retail store and using it continuously.

Today it's between 2 weeks for ML workloads to three months for the mid sized instances.

AWS made sense in big Corp when it would take you six months to get approval for buying the hardware and another six for the software. Today I'd only use it to do a prototype that I move on prem the second it looks like it will make it past one quarter.

Re: Why we use our own hardware

#374

Post by Amfy »

aquariusDue wrote: post #287
Beijinger wrote: post #207

I was told Fastmail is excellent, and I am not a big fan of gmail. Once locked out for good in gmail, your email and apps associated with it, are gone forever. Source? Personal experience. "A private inbox $60 for 12 months". I assume it is USD, not AU$ (AFAIK, Fastmail is based in Australia.) Still pricey. At https://www.infomaniak.com/ I can buy email service for an (in my case external) domain for 18 Euro a year a…

Personally I prefer Migadu and tend to recommend them to tech savvy people. Their admin panel is excellent and straightforward to use, prices are based on usage limits (amount of emails sent/received) instead of number of mailboxes. Migadu is just all around good, only downsides I can find are subjective. The fact that they're based in Switzerland and unless you're "good with computers" something like Fastmail will p…

Seems Migadu is hosted on OVH though? Huge red flag.. no control over infrastructure (think of Hetzner shutting down customers with little to no warning)

Re: Why we use our own hardware

#375

Post by 0xbadcafebee »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

You know how to set up a rock-solid remote hands console to all your servers, I take it? Dial-up modem to a serial console server, serial cables to all the servers (or IPMI on a segregated network and management ports). Then you deal with varying hardware implementations, OSes, setting that up in all your racks in all your colos.

Compare that to AWS, where there are 6 different kinds of remote hands, that work on all hardware and OSes, with no need for expertise, no time taken. No planning, no purchases, no shipment time, no waiting for remote hands to set it up, no diagnosing failures, etc, etc, etc...

That's just one thing. There's a thousand more things, just for a plain old VM. And the cloud provides way more than VMs.

The number of failures you can have on-prem is insane. Hardware can fail for all kinds of reasons (you must know this), and you have to have hot backup/spares, because otherwise you'll find out your spares don't work. Getting new gear in can take weeks (it "shouldn't" take that long, but there's little things like pandemics and global shortages on chips and disks that you can't predict). Power and cooling can go out. There's so many things that can (and eventually will) go wrong.

Why expose your business to that much risk, and have to build that much expertise? To save a few bucks on a server?

Re: Why we use our own hardware

#376

Post by jasode »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

>As the fortunes of AWS et al rose and rose and rose, I kept looking at their pricing at features and kept wondering what I was missing. They seemed orders of magnitude more expensive [...] To this day I still use bare metal servers for pretty much everything, [...] plain Linux, Bash, Perl, Python, and SSH, to handle everything cheaply

Your FastMail use case of (relatively) predictable server workload and product roadmap combined with agile Linux admins who are motivated to use close-to-bare-metal tools isn't an optimal cost fit for AWS. You're not missing anything and FastMail would have been overpaying for cloud.

Where AWS/GCP/Azure shine is organizations that need higher-level PaaS like managed DynamoDB, RedShift, SQS, etc that run on top of bare metal. Most non-tech companies with internal IT departments cannot create/operate "internal cloud services" that's on par with AWS.[1] Some companies like Facebook and Walmart can run internal IT departments with advanced capabilities like AWS but most non-tech companies can't. This means paying AWS' fat profit margins can actually be cheaper than paying internal IT salaries to "reinvent AWS badly" by installing MySQL, Kafka, etc on bare metal Linux. E.g. Netflix had their own datacenters in 2008 but a 3-day database outage that stopped them from shipping DVDs was one of the reasons they quit running their datacenters and migrated to AWS.[2] Their complex workload isn't a good fit for bare-metal Linux and bash scripts; Netflix uses a ton of high-level PaaS managed services from AWS.

If bare metal is the layer of abstraction the IT & dev departments are comfortable working at, then self-host on-premise, or co-lo, or Hetzner are all cheaper than AWS.

[1] https://web.archive.org/web/20160319022029/https://www.compu...

[2] https://media.netflix.com/en/company-blog/completing-the-net...

Re: Why we use our own hardware

#377

Post by Amfy »

goldeneye13_ wrote: post #5

Didn’t see this in the article, do they have multi az redundancy? I.e. if the entire raid goes up in flames what’s the recovery process?

I believe they replicate from NJ to WA (Seattle). At least that's something they spoke about many years ago.

Re: Why we use our own hardware

#378

Post by xorcist »

kasey_junk wrote: post #337
xorcist wrote: post #209

Earlier quoted context omitted.

Why do think that? I did something similar at a previous work for something bordering on 1k employees. User administration was done by modifying a yaml file in git. Nothing bad to say about it really. It sure beats point-and-click Active Directory any day of the week. Commit log handy for audits. If there are no externalities demanding anything else, I'd happily do it again.

There is nothing _wrong_ with it, and so long as you can prove that your offboarding is consistent and quick then feel free to use it. But a central system that uses the same identity/auth everywhere is much easier to keep consistent and fast. That’s why auditors and security professionals will harp on idp/sso solutions as some of the first things to invest in.

I found that the commit log made auditing on- and offboarding easier, not harder. Of course it won't help you if your process is dysfunctional. You still have to trigger the process somehow, which can be a problem in itself when growing from a startup, but once you do that it's smooth.

However git is a central system, a database if you will, where you can keep identities globally consistent. That's the whole point. In my experience, the reason people leave it is because you grow the need to interoperate with third party stuff which only supports AD or Okta or something. Should I get to grow past that phase myself I would feed my chosen IdM with that data instead.

Re: Why we use our own hardware

#379

Post by comprev »

lukevp wrote: post #351

To me, Cloud is all about the shift left of DevOps. It’s not a cost play. I’m a Dev Lead / Manager and have worked in both types of environments over the last 10 years. It’s immeasurable the velocity difference as far as system provisioning between the two approaches. In the hardware space, it took months to years to provision new machines or upgrade OSes. In the cloud, it’s a new terraform script and a CI deploy awa…

Our CI pipelines can spin up some seriously meaty hardware, run some very resource intensive tests, and destroy the infrastructure when finished.

Bonus points: they can do it with spot pricing to further lower the bill.

The cloud offers immense flexibility and empowers _developers_ to easily manage their own infrastructure without depending on other teams.

Speed of development is the primary reason $DayJob is moving into the cloud, while maintaining bare-metal for platforms which rarely change.

Post reply on HN