Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use our own hardware

fastmail.com

Post reply on HN
Print view View original

451–460 of 547 posts

Re: Why we use our own hardware

#451

Post by jedberg »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

In my 25 years, I've run some really big on-prem workloads and some of the biggest cloud loads (Sendmail.org and it's mail servers and Netflix streaming). Here is why I like the cloud:

Flexibility.

When Netflix wanted to start operating in Europe, we didn't have to negotiate datacenter space, order a bunch of servers, wait for racking and stacking, and all those other things. We just made an API call and had an entire stack built in Europe.

Same thing we we expanded to Asia.

It also saved us a ton of money, because our workload was about 3x peak to trough each day. We would scale up for peak, and scale down for trough.

We used on-prem for the parts where that made sense -- serving the actual video bits. Those were done on custom servers with a very stripped down FreeBSD optimized just for serving video (so optimized that we still used Akamai for images). But the part of the business that needed flexibility (control plane and interface) were all in AWS.

Why would a startup use the cloud? Both flexibility and ease. There aren't a lot of experts around that can configure a linux box from scratch anymore. And even if you can, you can't go from coded-up idea to production in five minutes like you can with the cloud. It would take you at least a few hours to set up the bare metal the first time.

Re: Why we use our own hardware

#452

Post by balex »

dehrmann wrote: post #229
UltraSane wrote: post #151

Earlier quoted context omitted.

"All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding," This is false. AWS infrastructure is vastly more secure than almost all company data centers. AWS has a rule that the same person cannot have logical access and physical access to the same storage device. Very few companies have enough IT people to have this rule. The AWS KMS is vastly m…

The other part is that when us-east-1 goes down, you can blame AWS, and a third of your customer's vendors will be doing the same. When you unplug the power to your colo rack while installing a new server, that's on you.

When there's little trust between the business and IT, both are incentivized to move to the cloud.

It's harder to build trust than the opposite.

Re: Why we use our own hardware

#453

Post by balex »

lukevp wrote: post #364
throwawaysxcd0 wrote: post #289

Earlier quoted context omitted.

OTOH, when your company's web site is down you can do something about it. When the CEO asks about it, you can explain why its offline and more importantly what is being done to bring it back. The equivalent situation for those who took a cloud based approach is often... ¯\_(ツ)_/¯

With the cloud, in a lot of cases you can have additional regions that incur very little cost as they scale dynamically with traffic. It’s hard to do that with on-prem. Also many AWS services come cross-AZ (AZ is a data center), so their arch is more robust than a single Colo server even if you’re in a single region.

Cross region from on-prem to the cloud for a website is easy. In fact, as long as you don't buy into "cloud native" ("cloud lock-in"?), it's probably more cost effective than two on-prem regions or two cloud regions.

Re: Why we use our own hardware

#454

Post by balex »

ttul wrote: post #329
johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

My firm belief after building a service at scale (tens of millions of end users, > 100K tps) is that AWS is unbeatable. We don’t even think about building our own infrastructure. There’s no way we could ever make it reliable enough, secure enough, and future-proof enough to ever pay back the cost difference. Something people neglect to mention when they tout their home grown cloud is that AWS spends significant cycle…

I've done it. It's nowhere as complicated as you make it seem. It definitely doesn't kill - no more than failing to manage your software tech debt. In fact, the latter is both harder keep up with and more risky, because it changes faster than the low level stuff, to support business needs.

With the cloud you have IT/DevOps deal only with scaling the software components of the infra. When doing on-prem they take on the physical layer as well. Do you have enough trust in them to scale the physical part where needed?

Re: Why we use our own hardware

#455

Post by balex »

watsocd wrote: post #424
ttul wrote: post #329

Earlier quoted context omitted.

My firm belief after building a service at scale (tens of millions of end users, > 100K tps) is that AWS is unbeatable. We don’t even think about building our own infrastructure. There’s no way we could ever make it reliable enough, secure enough, and future-proof enough to ever pay back the cost difference. Something people neglect to mention when they tout their home grown cloud is that AWS spends significant cycle…

...and power, backup power, HVAC, physical security...

Or buy colo space and they do it for you. It's not all cloud vs owning a datacenter - There's a thousand shades of ltgrey

Re: Why we use our own hardware

#456

Post by twotwotwo »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

> I've been doing some planning over the last couple of years on teaching a course on how to do all this

Yes! It's surprisingly common to hear it can't work, or can't scale or run reliably, when all that is done. Talking about how you've done it is great from that perspective.

Also, it's worth talking about what you gain, qualitatively! As this post mentions, your high-performance storage options are far better outside the cloud. People often mention egress, too. The appealing idea to me is using your extra flexibility to deploy better stuff, not saving a bit of cost.

Re: Why we use our own hardware

#457

Post by tiffanyh »

jedberg wrote: post #451
jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

In my 25 years, I've run some really big on-prem workloads and some of the biggest cloud loads (Sendmail.org and it's mail servers and Netflix streaming). Here is why I like the cloud: Flexibility. When Netflix wanted to start operating in Europe, we didn't have to negotiate datacenter space, order a bunch of servers, wait for racking and stacking, and all those other things. We just made an API call and had an entir…

When you say “cloud”, are you including old school web hosts that will rent you a dedicated server?

Like OVH, Hetzner or Hivelocity?

Because you can get some insane servers for like $300/month (eg brand new 5th gen Epyc 48-core / 0.5TB ram / lots of NVME) and globally available.

Re: Why we use our own hardware

#458

Post by chii »

browningstreet wrote: post #190
johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

Most companies severely understaff ops, infra, and security. Your talking points might be good but, in practice, won’t apply in many cases because of the intractability of that management mindset. Even when they should know better. I’ve worked at tech companies with hundreds of developers and single digit ops staff. Those people will struggle to build and maintain mature infra. By going cloud, you get access to matur…

It's the exact same reason why most companies don't just run their own power stations, and instead buy it from a power company.

Computation has become a utility these days - this includes the fat ISP lines and connectivity etc, not just the CPU and harddrives. These things have economies of scale that smaller companies cannot truly reach, and will pay a huge fixed cost if they want state of the art management, monitoring and redundancy. So unless you are a massive consumer, just like power stations, you really don't need nor want to build your own.

Re: Why we use our own hardware

#459

Post by dataflow »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

> As the fortunes of AWS et al rose and rose and rose, I kept looking at their pricing at features and kept wondering what I was missing.

How do the availability/fault tolerance compare? If one of your geographical locations gets knocked out (fire, flood, network cutoff, war, whatever) what will the user experience look like, vs. what can cloud providers provide?

Re: Why we use our own hardware

#460

Post by ForOldHack »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

" teaching a course on how to do all this..." Can you provide some notice of this so I can schedule my vacation time to fully participate? Let me know when registration is open.
Post reply on HN