Why we use our own hardware • Hacker News Board

Re: Why we use our own hardware

#381

Post by theideaofcoffee » Sun, Dec 22, 2024, 9:01 PM UTC

sgarland wrote: post #260

noprocrasted wrote: post #165

Earlier quoted context omitted.

> And running your own hardware is not incompatible with Kubernetes: on the contrary Kubernetes actually makes so much more sense on bare-metal hardware. On the cloud, I think the value prop is dubious - your cloud provider is already giving you VMs, why would you need to subdivide them further and add yet another layer of orchestration? Not to mention that you're getting 2010s-era performance on those VMs, so subdiv…

> Not to mention that you're getting 2010s-era performance on those VMs, so subdividing them is terrible from a performance point of view too. I was trying in vain to explain to our infra team a couple of weeks ago why giving my team a dedicated node of a newer instance family with DDR5 RAM would be beneficial for an application which is heavily constrained by RAM speed. People seem to assume that compute is homogeno…

I would wager that the same kind of people that were arguing against your request for a specific hardware config are the same ones in this comment section railing against any sort of self-sufficiency by hosting it yourself on hardware. All they know is cloud, all they know how to do is "ScAlE Up thE InStanCE!" when shit hits the fan. It's difficult to argue against that and make real progress. I understand your frustration completely.

Re: Why we use our own hardware

#382

Post by sufehmi » Sun, Dec 22, 2024, 9:03 PM UTC

goldeneye13_ wrote: post #5

Didn’t see this in the article, do they have multi az redundancy? I.e. if the entire raid goes up in flames what’s the recovery process?

https://www.fastmail.com/blog/throwback-security-confidentia...

Re: Why we use our own hardware

#383

Post by EdJiang » Sun, Dec 22, 2024, 9:20 PM UTC

I was a bit confused by the section on backups. How do they manage moving the data offsite with the on-premises backup servers? Wouldn’t that be a cost savings by going cloud?

Re: Why we use our own hardware

#384

Post by matt-p » Sun, Dec 22, 2024, 9:21 PM UTC

ksec wrote: post #353

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

>But everyone seemed to be flocking to them. To the point we have young Devs today that dont know what VPS and Colo ( Colocation) meant. Back to the article, I am surprised it was only a "A few years ago" Fastmail adopted SSD. Which certainly seems late in the cycle for the benefits of what SSD offers. Price for Colo on the order of $3000/2U/year. That is $125 /U/month.

Colo is typically sold on power not space, from your example you're either getting ripped off if it's for low power servers or massively undercharged for a 4xa100 machine

Re: Why we use our own hardware

#385

Post by drdaeman » Sun, Dec 22, 2024, 9:32 PM UTC

lukevp wrote: post #351

To me, Cloud is all about the shift left of DevOps. It’s not a cost play. I’m a Dev Lead / Manager and have worked in both types of environments over the last 10 years. It’s immeasurable the velocity difference as far as system provisioning between the two approaches. In the hardware space, it took months to years to provision new machines or upgrade OSes. In the cloud, it’s a new terraform script and a CI deploy awa…

> In the hardware space, it took months to years to provision new machines or upgrade OSes.

If it takes this long to manage a machine, I strongly suspect it means that when initially designing the system engineers had failed to account for those for some reason. Was that true in your case?

Back in late '00s until mid '10s, I worked for an ISP startup as a SWE. We had a few core machines (database, RADIUS server, self-service website, etc) - ugly mess TBH - initially provisioned and originally managed entirely by hand as we didn't knew any better back then. Naturally, maintaining those was a major PITA, so they sat on the same dated distro for years. That was before Ansible was a thing, and we haven't really heard about Salt or Chef before we started to feel the pains and started to search for solutions. Virtualization (OpenVZ, then Docker) helped to soften a lot of issues, making it significantly easier to maintain the components, but the pains from our original sins were felt for a long time.

But we also had a fleet of other machines, where we understood our issues with the servers enough to design new nodes to be as stateless as possible, with automatic rollout scripts for whatever we were able to automate. Provisioning a new host took only a few hours, with most time spent unpacking, driving, accessing the server room, and physically connecting things. Upgrades were pretty easy too - reroute customers to another failover node, write a new system image to the old one, reboot, test, re-route traffic back, done.

So it's not like self-owned bare metal is harder to manage - the lesson I learned is that one just gotta think ahead of time what the future would require. Same as the clouds, I guess, one has to follow best practices or they'll end up with crappy architectures that will be painful to rework. Just different set of practices, because of the different nature of the systems.

Re: Why we use our own hardware

#386

Post by immibis » Sun, Dec 22, 2024, 9:47 PM UTC

nine_k wrote: post #339

fulafel wrote: post #194

Earlier quoted context omitted.

OTOH: 1. big clouds are very lucrative targets for spooks, your data seem pretty likely to be hoovered up as "bycatch" (or maybe main catch depending on your luck) by various agencies and then traded around as currency 2. you never hear about security probems (incidents or exposure) in the platforms, there's no transparency 3. better than most coporate stuff is a low bar

If you don't want your data to be accessible to "various agencies", don't share it with corporations, full stop. Corporations are obliged by law to make it available to the agencies, and the agencies often overreach, while the corporations almost never mind the overreach. There are limitations for stuff like health or financial data, but these are not impenetrable barriers. I would just consider all your hosted data…

That depends where it's hosted and how it's encrypted. Cloud hosts can just reach into your RAM, but dedicated server hosts would need to provision that before deploying the server, and colocation providers would need to take your server offline to install it.

Re: Why we use our own hardware

#387

Post by switch007 » Sun, Dec 22, 2024, 9:48 PM UTC

0xbadcafebee wrote: post #375

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

You know how to set up a rock-solid remote hands console to all your servers, I take it? Dial-up modem to a serial console server, serial cables to all the servers (or IPMI on a segregated network and management ports). Then you deal with varying hardware implementations, OSes, setting that up in all your racks in all your colos. Compare that to AWS, where there are 6 different kinds of remote hands, that work on all…

This. All of this and more. I've got friends who worked for a hosting providers who over the years have echoed this comment. It's endless.

Re: Why we use our own hardware

#388

Post by jread » Sun, Dec 22, 2024, 9:53 PM UTC

motorest wrote: post #253

johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

> All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding,(...) This is where you lose all credibility. I'm going to focus on a single aspect: performance. If you're serving a global user base and your business, like practically all online businesses, is greatly impacted by performance problems, the only solution to a physics problem is to deplo…

If have a global user base, depending on your workload, a simple CDN in front of your hardware can often go a long ways with minimal cost and complexity.

Re: Why we use our own hardware

#389

Post by nine_k » Sun, Dec 22, 2024, 9:57 PM UTC

immibis wrote: post #386

nine_k wrote: post #339

Earlier quoted context omitted.

If you don't want your data to be accessible to "various agencies", don't share it with corporations, full stop. Corporations are obliged by law to make it available to the agencies, and the agencies often overreach, while the corporations almost never mind the overreach. There are limitations for stuff like health or financial data, but these are not impenetrable barriers. I would just consider all your hosted data…

That depends where it's hosted and how it's encrypted. Cloud hosts can just reach into your RAM, but dedicated server hosts would need to provision that before deploying the server, and colocation providers would need to take your server offline to install it.

Colocated / Dedicated is not Cloud, AFAICT. It's the "traditional hosting", not elastic / auto-scalable. You of course may put your own, highly tamper-proof boxes in a colocation rack, and be reasonably certain that any attempt to exfiltrate data from them won't be invisible to you.

By doing so, you share nothing with your hosting provider, you only rent rack space / power / connectivity.

Re: Why we use our own hardware

#390

Post by tokioyoyo » Sun, Dec 22, 2024, 10:02 PM UTC

sgarland wrote: post #202

tokioyoyo wrote: post #120

Earlier quoted context omitted.

I don’t think you’re correct. I’ve watched junior/mid-level engineers figure things out solely by working on the cloud and scaling things to a dramatic degree. It’s really not a rocket science.

I didn't say it's rocket science, nor that it's impossible to do without having practical server experience, only that it's more difficult. Take disks, for example. Most cloud-native devs I've worked with have no clue what IOPS are. If you saturate your disk, that's likely to cause knock-on effects like increased CPU utilization from IOWAIT, and since "CPU is high" is pretty easy to understand for anyone, the seeming…

> Most cloud-native devs

You might be making some generalizations from your personal experience. Since 2015, at all of my jobs, everything has been running on some sort of a cloud. I'm yet to meet a person who doesn't understand IOPS. If I was a junior (and from my experience, that's what they tend to do), I'd just google "slow X potential reasons". You'll most likely see some references to IOPS and continue your research from there.

We've learned all these things one way or another. My experience started around 2007ish when I was renting out cheap servers from some hosting providers. Others might be dipping their feet into readily available cloud-infrastructure, and learning it from that end. Both works.