Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use our own hardware

fastmail.com

Post reply on HN
Print view View original

511–520 of 547 posts

Re: Why we use our own hardware

#511

Post by gizzlon »

jandrewrogers wrote: post #274
johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

This trivializes some real issues. The biggest problem the cloud solves is hardware supply chain management. To realize the full benefits of doing your own build at any kind of non-trivial scale you will need to become an expert in designing, sourcing, and assembling your hardware. Getting hardware delivered when and where you need it is not entirely trivial -- components are delayed, bigger customers are given prior…

> managing hardware vendors, logistics, and delivery dates on an ongoing basis is a giant time suck

You can rent servers and it's still not cloud.

I'm pretty neutral and definitely see the value of cloud. But a lot of cloud proponents seem to lack, what to me, seems like basic knowledge.

Re: Why we use our own hardware

#512

Post by listenallyall »

jandrewrogers wrote: post #274
johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

This trivializes some real issues. The biggest problem the cloud solves is hardware supply chain management. To realize the full benefits of doing your own build at any kind of non-trivial scale you will need to become an expert in designing, sourcing, and assembling your hardware. Getting hardware delivered when and where you need it is not entirely trivial -- components are delayed, bigger customers are given prior…

> don't want to be bothered with any of these details

Isn't the job to be bothered with the details? 90% of employment for most people is doing shit you don't really want to be doing, but that's the job.

Re: Why we use our own hardware

#513

Post by everfrustrated »

samcat116 wrote: post #498
aragilar wrote: post #475

Earlier quoted context omitted.

I'm confused why you would even need AWS then (what's running on the VMs)? My impression is the standard compute (as in CPUs+RAM) isn't expensive, it's the storage (1 PB is less than half a rack physically now, comparing with the yearly prices listed), and so if you don't have much data, the value of on-prem isn't there.

For smaller shops I'd argue storage is the hardest part. I've done several OpenStack and baremetal K8s deployments on prem and the part that always stressed me out the most was storage. I'd happily pay a markup for that vs just about anything else that would be more economical to do on prem for smaller simpler workloads.

Also encrypted storage on AWS is so simple. Encrypted root file systems on prem is not easy.

Re: Why we use our own hardware

#514

Post by AndrewDavis »

karolzlot wrote: post #508
AndrewDavis wrote: post #447

Earlier quoted context omitted.

A few more than one, but it has been a lot less than when we were dealing with spinner. I think I requested about one or two replacements a year, a far cry from the one a week I was doing before.

Can I ask which brands / models of SSD are you using?

We didn’t replace all servers at once, it was progressive, therefore due to availability the models we used changed over time.

Our first batch of all nvme machines had SSDPE2KX080T8, they became harder to source and we moved to SSDPF2KX076T1.

With Intel no longer in the ssd business I believe we have some Micron MTFDKCC7T6TGH and MTFDKCC30T7TGR. And as mentioned in the blog post we've recently purchased some Solidigm D5-P5336 which are 61TB monsters.

Here's a fun related story. Our supplier had so much trouble finding SSDPE2KX080T8 that when we had exhausted our spares, I had to sync everything off a machine, tear it down and pull its drives for spares and rebuild it with the smaller SSDPF2KX076T1. Then we had lots of spares

Re: Why we use our own hardware

#515

Post by brongondwana »

rob_c wrote: post #510
brongondwana wrote: post #488

Earlier quoted context omitted.

I mean, I'm happy to have this argument. CEPH is content agnostic and that's fantastic most of the time. Cyrus replication is data aware, so it's not just replicating the data, it's doing integrity checking and data model consistency handling. Most of all, it's doing split brain recovery; which - if we wanted CP rather than AP then we wouldn't need, but that wasn't the original design. If I was redoing this from scra…

Do you honestly think CEPH isn't doing data consistency handling? I'll pay for your ticket to cephalocon if you'll speak to that effect(!) Split brain stuff only happens when you're splitting a single threaded task and put it back together. MDS in CEPH has this problem but that's so far into the weeds here as to be off topic. Again you're implementing something storage not in storage and taking any storage. Fine if y…

I do believe we are talking past each other here. Of course ceph does data consistency, but it sure doesn't assert that a modseq is monotonically increasing or that an mailbox/uidvalidity/uid triple doesn't change digest, because it's not data-model aware.

sigh

Re: Why we use our own hardware

#516

Post by 0xbadcafebee »

likeabatterycar wrote: post #400
0xbadcafebee wrote: post #375

Earlier quoted context omitted.

You know how to set up a rock-solid remote hands console to all your servers, I take it? Dial-up modem to a serial console server, serial cables to all the servers (or IPMI on a segregated network and management ports). Then you deal with varying hardware implementations, OSes, setting that up in all your racks in all your colos. Compare that to AWS, where there are 6 different kinds of remote hands, that work on all…

> The number of failures you can have on-prem is insane. Hardware can fail for all kinds of reasons (you must know this) Cloud vendors are not immune from hardware failure. What do you think their underlying infrastructure runs on, some magical contraption made from Lego bricks, Swiss chocolate, and positive vibes? It's the same hardware, prone to the same failures. You've just outsourced worrying about it.

The hardware is prone to the same failures, but the customers rarely experience them, because they handle it for you. EBS means never worrying about disks. S3 means never worrying about objects. EC2 ASG means never worrying about failed machines/VMs. Multi-AZ means never worrying about an entire datacenter going down.

Yes, you pay someone else to worry about it. That's kinda the whole idea.

Re: Why we use our own hardware

#517

Post by inkyoto »

noprocrasted wrote: post #157
jeffbee wrote: post #99

Earlier quoted context omitted.

In what sense is AWS "bare cloud"? S3, DynamoDB, Lambda, ECS?

How do you configure S3 access control? You need to learn & understand how their IAM works. How do you even point a pretty URL to a lambda? Last time I looked you need to stick an "API gateway" in front (which I'm sure you also get nickel & dimed for). How do you go from "here's my git repo, deploy this on Fargate" with AWS? You need a CI pipeline which will run a bunch of awscli commands. And I'm not even talking ab…

How does one install and run Linux/BSD/another UNIX? One needs to learn and understand how a UNIX works.

The essence of the complaint that one has to have the knowledge of something before that something can be used. It seems like a reasonable expectation for just about anything in life.

(The API gateway in AWS is USD 2.35 for 10 million 32 kB requests, a Lambda can have its own private URL if required and Fargate does not deploy Git repos, it runs Docker images.)

Re: Why we use our own hardware

#518

Post by marcosdumay »

sgarland wrote: post #153
floating-io wrote: post #114

Earlier quoted context omitted.

"Vendor problems" is a red herring, IMO; you can have those in the cloud, too. It's been my experience that those who can build good, reliable, high-quality systems, can do so either in the cloud or on-prem, generally with equal ability. It's just another platform to such people, and they will use it appropriately and as needed. Those who can only make it work in the cloud are either building very simple systems (whi…

> or are building a house of cards that will eventually collapse (or just cost them obscene amounts of money to keep on life support) Ding ding ding. It's this. > The whole CapEx vs. OpEx issue drives me batty Seconded. I can't help but feel like it's not just a "I don't understand money" thing, but more of a "the way Wall Street assigns value is fundamentally broken." Spending $100K now, once, vs. spending $25K/mont…

> Spending $100K now, once, vs. spending $25K/month indefinitely does not take a genius to figure out.

If you multiply your month payment for 1/i, where i is the interest rate your business can get, you will get how much of up-front money it's worth.

... that is, until next month, when the interest rate will change, a fact that always catches everyone by surprise, and you'll need to rush to fix your cash-flow.

So, yeah, I don't understand that either. Somehow, despite neither of us understanding how it can possibly work, it seems to fail to work empirically too, adding a huge amount of instability to companies.

That is, unless you decide to look at it from the perspective of executive bonuses, that are capped to 0, but can grow indefinitely. So instability is the point.

Re: Why we use our own hardware

#519

Post by crossroadsguy »

mariusor wrote: post #273
Beijinger wrote: post #207

I was told Fastmail is excellent, and I am not a big fan of gmail. Once locked out for good in gmail, your email and apps associated with it, are gone forever. Source? Personal experience. "A private inbox $60 for 12 months". I assume it is USD, not AU$ (AFAIK, Fastmail is based in Australia.) Still pricey. At https://www.infomaniak.com/ I can buy email service for an (in my case external) domain for 18 Euro a year a…

My suggestion would be to try Purelymail. They don't offer much in the way of a web interface to email, but if you bring your own client, it's a very good provider. I'm paying something like $10 per year for multiple domains with multiple email addresses (though with little traffic). I've been using them for about 5 years and I had absolutely no issues.

Purelymail is just one person show. May that one person live long and prosper, but I am not putting my faith or email in that business.

Re: Why we use our own hardware

#520

Post by ldng »

jedberg wrote: post #464
tiffanyh wrote: post #457

Earlier quoted context omitted.

When you say “cloud”, are you including old school web hosts that will rent you a dedicated server? Like OVH, Hetzner or Hivelocity? Because you can get some insane servers for like $300/month (eg brand new 5th gen Epyc 48-core / 0.5TB ram / lots of NVME) and globally available.

Those could count. But you'll still end up having to do some linux admin, which a lot of people can't do anymore. The whole point is that the closer you can get to "write code, run code", the faster you can launch and innovate.

Linux admin still exists. Except that they are better paid than ever at cloud provider. What you're describing is more payroll flexibility than technical.
Post reply on HN