Live data from Hacker News

Why we use our own hardware

fastmail.com

121–130 of 547 posts

Re: Why we use our own hardware

#121

Earlier quoted context omitted.

Does it? I've seen outages around "Sorry, us-west_carolina-3 is down". AWS is particularly good at keeping you aware of their datacenters.

if you see that you are doing it wrong :)

AWS has had multiple outages which were caused by a single AZ failing.

Re: Why we use our own hardware

#122
post #83

Earlier quoted context omitted.

Cloud expands the capabilities of what one team can manage by themselves, enabling them to avoid a huge amount of internal politics. This is worth astronomical amounts of money in big corps.

I’m not convinced this is entirely true. The upfront cost if you don’t have the skills, sure – it takes time to learn Linux administration, not to mention management tooling like Ansible, Puppet, etc. But once those are set up, how is it different? AWS is quite clear with their responsibility model that you still have to tune your DB, for example. And for the setup, just as there are Terraform modules to do everythin…

You are focusing on technology. And sure of course you can get most of the benefits of AWS a lot cheaper when self-hosting.

But when you start factoring internal processes and incompetent IT departments, suddenly that's not actually a viable option in many real-world scenarios.

Re: Why we use our own hardware

#123
post #83

Earlier quoted context omitted.

Cloud expands the capabilities of what one team can manage by themselves, enabling them to avoid a huge amount of internal politics. This is worth astronomical amounts of money in big corps.

I’m not convinced this is entirely true. The upfront cost if you don’t have the skills, sure – it takes time to learn Linux administration, not to mention management tooling like Ansible, Puppet, etc. But once those are set up, how is it different? AWS is quite clear with their responsibility model that you still have to tune your DB, for example. And for the setup, just as there are Terraform modules to do everythin…

I think this is only true for teams and apps of a certain size.

I've worked on plenty of teams with relatively small apps, and the difference between:

1. Cloud: "open up the cloud console and start a VM"

2. Owned hardware: "price out a server, order it, find a suitable datacenter, sign a contract, get it racked, etc."

Is quite large.

#1 is 15 minutes for a single team lead.

#2 requires the team to agree on hardware specs, get management approval, finance approval, executives signing contracts. And through all this you don't have anything online yet for... weeks?

If your team or your app is large, this probably all averages out in favor of #2. But small teams often don't have the bandwidth or the budget.

Re: Why we use our own hardware

#124
post #83

Earlier quoted context omitted.

Cloud expands the capabilities of what one team can manage by themselves, enabling them to avoid a huge amount of internal politics. This is worth astronomical amounts of money in big corps.

I’m not convinced this is entirely true. The upfront cost if you don’t have the skills, sure – it takes time to learn Linux administration, not to mention management tooling like Ansible, Puppet, etc. But once those are set up, how is it different? AWS is quite clear with their responsibility model that you still have to tune your DB, for example. And for the setup, just as there are Terraform modules to do everythin…

You can get pretty far without any of that fancy stuff. You can get plenty done by using parallel-ssh and then focusing on the actual thing you develop instead of endless tooling and docker and terraform and kubernetes and salt and puppet and ansible. Sure, if you know why you need them and know what value you get from them OK. But many people just do it because it's the thing to do...

Re: Why we use our own hardware

#125

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

> What's particularly fascinating to me, though, is how some people are so pro-cloud that they'd argue with a writeup like this with silly cloud talking points. I’m sure I’ll be downvoted to hell for this, but I’m convinced that it’s largely their insecurities being projected. Running your own hardware isn’t tremendously difficult, as anyone who’s done it can attest, but it does require a much deeper understanding of…

> So for someone who may well be quite skilled at K8s administration ...

And running your own hardware is not incompatible with Kubernetes: on the contrary. You can fully well have your infra spin up VMs and then do container orchestration if that's your thing.

And part your hardware monitoring and reporting tool can work perfectly fine from containers.

Bare metal -> Hypervisor -> VM -> container orchestration -> a container running a "stateless" hardware monitoring service. And VMs themselves are "orchestrated" too. Everything can be automated.

Anyway say a harddisk being to show errors? Notifications being sent (email/SMS/Telegram/whatever) by another service in another container, dashboard shall show it too (dashboards are cool).

Go to the machine once the spare disk as already been resilvered, move it where the failed disk was, plug in a new disk that becomes the new spare.

Boom, done.

I'm not saying all self-hosted hardware should do container orchestration: there are valid use cases for bare metal too.

But something as to be said about controlling everything on your own infra: from the bare metal to the VMs to container orchestration. To even potentially your own IP address space.

This is all within reach of an individual, both skill-wise and price-wise (including obtaining your own IP address space). People who drank the cloud kool-aid should ponder this and wonder how good their skills truly are if they cannot get this up and working.

Re: Why we use our own hardware

#126
post #87

Earlier quoted context omitted.

Cloud solves one problem quite well: Geographic redundancy. It's extremely costly with on-prem.

Does it? I've seen outages around "Sorry, us-west_carolina-3 is down". AWS is particularly good at keeping you aware of their datacenters.

It can be useful. I run a latency sensitive service with global users. A cloud lets me run it in 35 locations dealing with one company only. Most of those locations only have traffic to justify a single, smallish, instance.

In the locations where there's more traffic, and we need more servers, there are more cost effective providers, but there's value in consistency.

Elasticity is nice too, we doubled our instance count for the holidays, and will return to normal in January. And our deployment style starts a whole new cluster, moves traffic, then shuts down the old cluster. If we were on owned hardware, adding extra capacity for the holidays would be trickier, and we'd have to have a more sensible deployment method. And the minimum service deployment size would probably not be a little quad processor box with 2GB ram.

Using cloud for the lower traffic locations and a cost effective service for the high traffic locations would probably save a bunch of money, but add a lot of deployment pain. And a) it's not my decision and b) the cost difference doesn't seem to be quite enough to justify the pain at our traffic levels. But if someone wants to make a much lower margin, much simpler service with lots of locations and good connectivity, be sure to post about it. But, I think the big clouds have an advantage in geographic expansion, because their other businesses can provide capital and justification to build out, and high margins at other locations help cross subsidize new locations when they start.

Re: Why we use our own hardware

#127
post #117

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

There was a time when cloud was significantly cheaper then owning. I'd expect that there are people who moved to the cloud then, and over time started using services offered by their cloud provider (e.g., load balancers, secret management, databases, storage, backup) instead of running those services themselves on virtual machines, and now even if it would be cheaper to run everything on owned servers they find it wo…

That was never true except in the case that the required hardware resources were significantly smaller than a typical physical machine.

Re: Why we use our own hardware

#128
post #109

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

I think part of it was a way for dev teams to get an infra team that was not empowered to say no. Plus organizational theory, empire building, etc.

Yep. I had someone tell me last week that they didn't want a more rigid schema because other teams rely on it, and anything adding "friction" to using it would be poorly received.

As an industry, we are largely trading correctness and performance for convenience, and this is not seen as a negative by most. What kills me is that at every cloud-native place I've worked at, the infra teams were both responsible for maintaining and fixing the infra that product teams demanded, but were not empowered to push back on unreasonable requests or usage patterns. It's usually not until either the limits of vertical scaling are reached, or a SEV0 occurs where these decisions were the root cause does leadership even begin to consider changes.

Re: Why we use our own hardware

#129
post #99

Earlier quoted context omitted.

That argument makes sense for PaaS services like the ones you mention. But for bare "cloud" like AWS, I'm not convinced it is saving any effort, it's merely swapping one kind of complexity with another. Every place I've been in had full-time people messing with YAML files or doing "something" with the infrastructure - generally trying to work around the (self-inflicted) problems introduced by their cloud provider - w…

In what sense is AWS "bare cloud"? S3, DynamoDB, Lambda, ECS?

EC2

Re: Why we use our own hardware

#130

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

Dunno man, it's really really easy to set up an S3 and use it to share datasets for users authorized with IAM....

And IAM and other cloud security and management considerations is where the opex/capex and capability argument can start to break down. Turns out, the "cloud" savings comes from not having capabilities in house to manage hardware. Sometimes, for most businesses, you want some of that lovely reliability.

(In short, I agree with you, substantially).

Like code. It is easy to get something basic up, but substantially more resources are needed for non-trivial things.

Post reply on HN