Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use our own hardware

fastmail.com

Post reply on HN
Print view View original

531–540 of 547 posts

Re: Why we use our own hardware

#531

Post by noprocrasted »

inkyoto wrote: post #517
noprocrasted wrote: post #157

Earlier quoted context omitted.

How do you configure S3 access control? You need to learn & understand how their IAM works. How do you even point a pretty URL to a lambda? Last time I looked you need to stick an "API gateway" in front (which I'm sure you also get nickel & dimed for). How do you go from "here's my git repo, deploy this on Fargate" with AWS? You need a CI pipeline which will run a bunch of awscli commands. And I'm not even talking ab…

How does one install and run Linux/BSD/another UNIX? One needs to learn and understand how a UNIX works. The essence of the complaint that one has to have the knowledge of something before that something can be used. It seems like a reasonable expectation for just about anything in life. (The API gateway in AWS is USD 2.35 for 10 million 32 kB requests, a Lambda can have its own private URL if required and Fargate do…

> The essence of the complaint that one has to have the knowledge of something before that something can be used

My point was to disprove that "cloud" is simpler than conventional sysadmin - it is not, and it involves similar effort, complexity and manpower requirements.

Re: Why we use our own hardware

#532

Post by UltraSane »

lukevp wrote: post #364
throwawaysxcd0 wrote: post #289

Earlier quoted context omitted.

OTOH, when your company's web site is down you can do something about it. When the CEO asks about it, you can explain why its offline and more importantly what is being done to bring it back. The equivalent situation for those who took a cloud based approach is often... ¯\_(ツ)_/¯

With the cloud, in a lot of cases you can have additional regions that incur very little cost as they scale dynamically with traffic. It’s hard to do that with on-prem. Also many AWS services come cross-AZ (AZ is a data center), so their arch is more robust than a single Colo server even if you’re in a single region.

Being able to choose from so many different Availability Zones in so many different regions is one of the best things about AWS. Combined with sophisticated routing strategies that Route 53 supports allows for some very effective designs.

Re: Why we use our own hardware

#533

Post by benterix »

lobsterthief wrote: post #528
benterix wrote: post #358

Earlier quoted context omitted.

> As the fortunes of AWS et al rose and rose and rose, I kept looking at their pricing at features and kept wondering what I was missing. You are not the only one. There are several factors at play but I believe one of the strongest today is the generational divide: the people lost the ability to manage their own infra or don't know it well enough to do it well so it's true when they say "It's too much hassle". I say…

It’s also human capital/resource allocation. We thought about spinning up our own servers at my last gig; we had the talent in house but that talent was busy building the product, not managing servers. I suppose it depends on what your need is as well.

I see your point but my perspective on this shifted over the years. Whatever infra you set up, whether it's the public cloud or on prem, there is always the initial cost (starting with a simple account for small orgs, a landing zone for larger ones etc.) and this applies to every service, it's just registered in the books in a slightly different way. For example, whn you look at my Jira tickets, on prem we're patchng servers, and in the cloud we're usually updating container images. These two are not that different and you need to set aside some time for that. It's the same with upgrading Postgres on prem and RDS Postgres between major versions - you need to arrange the service window with product teams, do the migration on lower layers first and if all goes well you move on to prod.

Of course, many infra activities take less time in the public cloud. E.g. control plane maintenance and upgrades on EKS are managed by AWS and are mostly painless so you never worry about stuff like etcd. On the other hand, there is a ton of stuff you need to know anyway to operate AWS in a proficient and safe way so I'm not convinced the difference is that huge today.

Re: Why we use our own hardware

#534

Post by AtlasBarfed »

blackqueeriroh wrote: post #495
AtlasBarfed wrote: post #311

Earlier quoted context omitted.

It's now been two years since I used KMS, but at the time it seemed little more than S3 API interface with Twitter size limitations Fundamentally why would KMS be more secure than S3 anyway? Both ultimately have the same fundamental security requirements and do the same thing. So the big whirlydoo is KMS has hardware keygen. im sorry, that sounds like something almost guaranteed to have nsa backdoor, or has so much n…

I guess since this is Hacker News, I shouldn’t be surprised that there are a bunch of commenters who are absolutely certain they and their random colo provider will do a better job of defeating the almighty NSA than AWS. You won’t even know when they serve your Colo provider with a warrant under gag order, and I’m certain they’ll be able to bypass your own “tamper-proof” protections.

Soo..... you're saying that KMS hardware key generation isn't that great anyway...

so, again, why bother with KMS? What does it offer?

My point about the hardware was asking why KMS hardware key generation has any real value vs a software generated key, and then why bother with KMS and its limited secret size, and you access KMS with a policy/security user or role that can be used equally to lock down S3?

What is the value of KMS?

Re: Why we use our own hardware

#535

Post by crossroadsguy »

mariusor wrote: post #526
crossroadsguy wrote: post #519

Earlier quoted context omitted.

Purelymail is just one person show. May that one person live long and prosper, but I am not putting my faith or email in that business.

Why do you need to put faith in them? Switching email providers is just a DNS change away, and email messages can be stored locally - actually it's encouraged to do so.

You do have a point. But I still do not feel comfortable. Besides I can't envisage changing providers whenever I face trouble and it gets unresolved in a timely manner which is what I assume in the current setup. Pinboard is another example. Anyway, it kind of doesn't work for me.

Re: Why we use our own hardware

#536

Post by causal »

jasode wrote: post #376
jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

>As the fortunes of AWS et al rose and rose and rose, I kept looking at their pricing at features and kept wondering what I was missing. They seemed orders of magnitude more expensive [...] To this day I still use bare metal servers for pretty much everything, [...] plain Linux, Bash, Perl, Python, and SSH, to handle everything cheaply Your FastMail use case of (relatively) predictable server workload and product roa…

Right, AWS rarely saves on hardware/hosting costs, it saves developer-hours. Especially if you're a fast-moving organization that rapidly changing hardware needs, something like AWS gives you agility.

That said, most organizations are not nearly so agile as they'd like to believe and would probably be better off paying for something inflexible and cheap.

Re: Why we use our own hardware

#537

Post by veidr »

veidr wrote: post #147
veidr wrote: post #142

"WHY we use our own hardware..." The why is is the interesting part of this article.

I take that back; this is (to me)t he most interesting part: "Although we’ve only ever used datacenter class SSDs and HDDs failures and replacements every few weeks were a regular occurrence on the old fleet of servers. Over the last 3+ years, we’ve only seen a couple of SSD failures in total across the entire upgraded fleet of servers. This is easily less than one tenth the failure rate we used to have with HDDs."

I wanted to revisit this after checking my own anecdata. (But based on logfiles not just like recollections.)

I've had a ZFS system or some sort for about 10 years, and before that I had proprietary RAID chassis like Pegasus2 and Synology etc.

I can't quite say how many drives I have used, because my records are not that good. But maybe its like 100 drives since 2008. Maybe 150. Less than 200.

I had over 10 HDD devices fail (probably 13, confidence of like 90%).

I've only ever had 1 SSD fail.

I've also used the absolute cheapest shite SSDs.

I suspect the failure modes tend to be

- hard disks fail whenever the fuck, who knows

- SSDs fail in the beginning or end of their reasonable service life

P.S. With ZFS though, you don't really care if/when they fail. I've so far (knock on wood) never lost any data with a ZFS config with >1 disk redundancy and reasonable backups.

Re: Why we use our own hardware

#539

Post by bob_theslob646 »

jph00 wrote: post #344

The original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months…

I would gladly take your course if you offered it.

Re: Why we use our own hardware

#540

Post by inkyoto »

noprocrasted wrote: post #531
inkyoto wrote: post #517

Earlier quoted context omitted.

How does one install and run Linux/BSD/another UNIX? One needs to learn and understand how a UNIX works. The essence of the complaint that one has to have the knowledge of something before that something can be used. It seems like a reasonable expectation for just about anything in life. (The API gateway in AWS is USD 2.35 for 10 million 32 kB requests, a Lambda can have its own private URL if required and Fargate do…

> The essence of the complaint that one has to have the knowledge of something before that something can be used My point was to disprove that "cloud" is simpler than conventional sysadmin - it is not, and it involves similar effort, complexity and manpower requirements.

I will have to disagree on that.

Cloud is simpler than conventional sysadmin, once its foundational principles are understood and the declarative approach to the cloud architecture is adopted. If I want to run a solution, cloud gives me just that – a platform that simply runs my solution and abstracts the sysadmin ugliness away.

I have experienced both sides, including UNIX kernel and system programming, and I don't want to even think about sysadmin unless I want to tinker with a UNIX box on a weekend as a leisure activity.

Post reply on HN