Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use our own hardware

fastmail.com

Post reply on HN
Print view View original

501–510 of 547 posts

Re: Why we use our own hardware

#501

Post by bob1029 »

samcat116 wrote: post #498
aragilar wrote: post #475

Earlier quoted context omitted.

I'm confused why you would even need AWS then (what's running on the VMs)? My impression is the standard compute (as in CPUs+RAM) isn't expensive, it's the storage (1 PB is less than half a rack physically now, comparing with the yearly prices listed), and so if you don't have much data, the value of on-prem isn't there.

For smaller shops I'd argue storage is the hardest part. I've done several OpenStack and baremetal K8s deployments on prem and the part that always stressed me out the most was storage. I'd happily pay a markup for that vs just about anything else that would be more economical to do on prem for smaller simpler workloads.

This is it for me too. EBS is a bigger deal than the EC2 instances themselves.

Re: Why we use our own hardware

#502

Post by collingreen »

BrandoElFollito wrote: post #493
ForOldHack wrote: post #461

Earlier quoted context omitted.

"buying the hardware from a retail store." Never buy wholesale and never develop on immature hardware, I have seen c** with multiple 9 y.o. dev servers. I could shorten the ROI to less than 6 months.

What is c*? (seriously, I am not a native speaker and cannot turn the stars into a word that makes sense)

No worries, I AM a native speaker and I can't figure out the stars OR the specific parsing of that comment

Re: Why we use our own hardware

#503

Post by samcat116 »

swozey wrote: post #275
johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

I have about 30 years as a linux eng, starting with openbsd and have spent a LOT of time with hardware building webhosts and CDNs until about 2020 where my last few roles have been 100% aws/gcloud/heroku. I love building the cool edge network stuff with expensive bleeding edge hardware, smartnics, nvmeOF, etc but its infinitely more complicated and stressful than terraforming an AWS infra. Every cluster I set up I ha…

This 1000%. There are so many cool networking/virtualization/hardware things I love dealing with. But the stress of doing ceph upgrades isn't the right trade off usually.

Re: Why we use our own hardware

#505

Post by jread »

0xbadcafebee wrote: post #472
jread wrote: post #398

Earlier quoted context omitted.

> Hardware can fail for all kinds of reasons Complex cloud infra can also fail for all kinds of reasons, and they are often harder to troubleshoot than a hardware failure. My experience with server grade hardware in a reliable colo with a good uplink is it's generally an extremely reliable combination.

And my experience is the opposite, on both counts. I guess it's moot because two anecdotes cancel each other out? Cloud VMs fail from either the instance itself not coming back online, or an EBS failure, or some other az-wide or region-wide failure that affects networking or control plane. It's very rare, but I have seen it happen - twice, across more than a thousand AWS accounts in 10 years. But even when it does ha…

There is lot of statistical and empirical data on this topic - MTBF estimates from vendors (typically 100k - 1m+ hours), Backblaze and Google drive failure data (~1-2% annual failure rate), IEEE and others. With N+1 redundancy (backup servers/RAID + spare drives) and proper design and change control processes, operational failures should be very rare.

With cloud hardware issues are just the start - yes you MUST "plan for failure", leveraging load balancers, auto scaling, cloudwatch, and dozens of other proprietary dials and knobs. However, you must also consider control plane, quotas, capacity, IAM, spend, and other non-hardware breaking points.

You're autoscaling isn't working - is the AZ out of capacity, did you hit a quota limit, run out of IPv4s, or was an AMI inadvertently removed? Your instance is unable to write to S3 - is the metadata service being flakey (for your IAM role), or is it due to an IAM role / S3 policy change? Your Lambda function is failing - did it hit a timeout, or exhaust the (512MB) temp storage? Need help diagnosing an issue - what is your paid support tier - submit a ticket and we'll get back to you sometime in the 24 hours.

Re: Why we use our own hardware

#506

Post by beaugunderson »

ylee wrote: post #483
beaugunderson wrote: post #425

I don't trust anything from fastmail after they bought pobox and forced me onto their new service which fails at the one thing pobox did well--forwarding email. They also refused to give me a refund (prorated or not) for removing the product I was using and substituting a defective one.

What problems have you had? I also came over from pobox and thought that the transition was quite straightforward.

Anything erroneously marked as spam can not be released to the forwarding address—-meaning they fail at their one job, forwarding email. Pobox had a great interface for quickly releasing messages to the forwarding address.

Re: Why we use our own hardware

#507

Post by neeeeeeal »

What not many people talk about in the comments is how the hardware route is fairly stacked against smaller players. Large enterprises buy the same hardware as small and midsize businesses at a fraction of the cost, which significantly impacts the economics of this decision. Even if you have the capability and desire, if each server costs your business double what an enterprise would pay, it becomes less attractive pretty quickly.

Re: Why we use our own hardware

#508

Post by karolzlot »

AndrewDavis wrote: post #447
brongondwana wrote: post #410

Earlier quoted context omitted.

My beard isn't entirely grey yet! The new NVMe drives we've only had for a few years, but so far there's only been a single failure across the whole fleet, and we keep spares in stock. It's been very reliable, not like the weeks back in (hmm, 2006? 2007?) the ancient past, when we were losing 15kRPM velociraptors every other day. They had a firmware fault and we eventually got an update which made them reliable, but…

A few more than one, but it has been a lot less than when we were dealing with spinner. I think I requested about one or two replacements a year, a far cry from the one a week I was doing before.

Can I ask which brands / models of SSD are you using?

Re: Why we use our own hardware

#509

Post by immibis »

doublerabbit wrote: post #487
nine_k wrote: post #389

Earlier quoted context omitted.

Colocated / Dedicated is not Cloud, AFAICT. It's the "traditional hosting", not elastic / auto-scalable. You of course may put your own, highly tamper-proof boxes in a colocation rack, and be reasonably certain that any attempt to exfiltrate data from them won't be invisible to you. By doing so, you share nothing with your hosting provider, you only rent rack space / power / connectivity.

And this is why I colocate, because all the data that hits my server is my data. Sure I do have an AUP/T&C but without proper warrant no one is allowed to touch my server. Case is monitored if it's opened. Encrypted on start-up, USB disabled. I just wished I had my own /24.

At least you can get your own /48, at least if you're under RIPE.

You should only do it if you expect to multihome though, or you're doing some experimentation that absolutely needs a PI address. Please don't pollute the default-free zone just for no reason.

Re: Why we use our own hardware

#510

Post by rob_c »

brongondwana wrote: post #488
rob_c wrote: post #480

Earlier quoted context omitted.

No, let's be honest. CEPH is the only solution for data management at this scale (sub to few PB). The solution which is independent of application or workload. The market share, fact IBM is moving people off other projects internally for this, and the massive backing shows this. Yes you can have all or a bunch of these features like failure domains via other routes/products but none have all of the stuff together in…

I mean, I'm happy to have this argument. CEPH is content agnostic and that's fantastic most of the time. Cyrus replication is data aware, so it's not just replicating the data, it's doing integrity checking and data model consistency handling. Most of all, it's doing split brain recovery; which - if we wanted CP rather than AP then we wouldn't need, but that wasn't the original design. If I was redoing this from scra…

Do you honestly think CEPH isn't doing data consistency handling? I'll pay for your ticket to cephalocon if you'll speak to that effect(!)

Split brain stuff only happens when you're splitting a single threaded task and put it back together. MDS in CEPH has this problem but that's so far into the weeds here as to be off topic.

Again you're implementing something storage not in storage and taking any storage. Fine if you want to do it that way, but talk about _that_ not hecking ZFS being mah saviour. (Btw daily driving and love that too but an email provider _relying_ on it should raise eye brows)...

Post reply on HN