Why we use our own hardware • Hacker News Board

Re: Why we use our own hardware

#241

Post by icedchai » Sun, Dec 22, 2024, 5:10 PM UTC

sgarland wrote: post #226

icedchai wrote: post #132

Earlier quoted context omitted.

Writing piles of IaC code like Terraform and CloudFormation is also a PITA and a distraction from focusing on your core product. PaaS is probably the way to go for small apps.

A small app (or a larger one, for that matter) can quite easily run on infra that's instantiated from canned IaC, like TF AWS Modules [0]. If you can read docs, you should be able to quite trivially get some basic infra up in a day, even with zero prior experience managing it. [0]: https://github.com/terraform-aws-modules

Yes, I've used several of these modules myself. They save tons of time! Unfortunately, for legacy projects, I inherited a bunch of code from individuals that built everything "by hand" then copy-pasted everything. No re-usability.

Re: Why we use our own hardware

#242

Post by warner25 » Sun, Dec 22, 2024, 5:11 PM UTC

tylerchurch wrote: post #123

sgarland wrote: post #103

Earlier quoted context omitted.

I’m not convinced this is entirely true. The upfront cost if you don’t have the skills, sure – it takes time to learn Linux administration, not to mention management tooling like Ansible, Puppet, etc. But once those are set up, how is it different? AWS is quite clear with their responsibility model that you still have to tune your DB, for example. And for the setup, just as there are Terraform modules to do everythin…

I think this is only true for teams and apps of a certain size. I've worked on plenty of teams with relatively small apps, and the difference between: 1. Cloud: "open up the cloud console and start a VM" 2. Owned hardware: "price out a server, order it, find a suitable datacenter, sign a contract, get it racked, etc." Is quite large. #1 is 15 minutes for a single team lead. #2 requires the team to agree on hardware s…

You gave me flashbacks to a far worse bureaucratic nightmare with #2 in my last job.

I supported an application with a team of about three people for a regional headquarters in the DoD. We had one stack of aging hardware that was racked, on a handshake agreement with another team, in a nearby facility under that other team's control. We had to periodically request physical access for maintenance tasks and the facility routinely lost power, suffered local network outages, etc. So we decided that we needed new hardware and more of it spread across the region to avoid the shaky single-point-of-failure.

That began a three year process of: waiting for budget to be available for the hardware / license / support purchases; pitching PowerPoints to senior management to argue for that budget (and getting updated quotes every time from the vendors); working out agreements with other teams at new facilities to rack the hardware; traveling to those sites to install stuff; and working through the cybersecurity compliance stuff for each site. I left before everything was finished, so I don't know how they ultimately dealt with needing, say, someone to physically reseat a cable in Japan (an international flight away).

Re: Why we use our own hardware

#243

Post by dehrmann » Sun, Dec 22, 2024, 5:11 PM UTC

johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

The real cost wins of self-hosted are that anything using new hardware becomes an ordeal, and engineers won't use high-cost, value-added services. I agree that there's often too little restraint in cloud architectures, but if a business truly believes in a project, it shouldn't be held up for six months waiting for server budget with engineers spending doing ops work to get three nines of DB reliability.

There is a size where self-hosting makes sense, but it's much larger than you think.

Re: Why we use our own hardware

#244

Post by mark242 » Sun, Dec 22, 2024, 5:12 PM UTC

johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

I'm curious about what "reasonable amount of hosting" means to you, because from my experience, as your internal network's complexity goes up, it's far better for your to move systems to a hyperscaler. The current estimate is >90% of Fortune 500 companies are cloud-based. What is it that you know that they don't?

Re: Why we use our own hardware

#245

Post by shortsunblack » Sun, Dec 22, 2024, 5:12 PM UTC

swiftcoder wrote: post #198

johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

> All the pro-cloud talking points... don't persuade anyone with any real technical understanding This is a very engineer-centric take. The cloud has some big advantages that are entirely non-technical: - You don't need to pay for hardware upfront. This is critical for many early-stage startups, who have no real ability to predict CapEx until they find product/market fit. - You have someone else to point the SOC2/HIP…

You can over-provision your own baremetal resources 20x and it will be still cheaper than cloud. The capex talking point is just that, a talking point.

Re: Why we use our own hardware

#246

Post by sanderjd » Sun, Dec 22, 2024, 5:14 PM UTC

xorcist wrote: post #185

tylerchurch wrote: post #123

Earlier quoted context omitted.

I think this is only true for teams and apps of a certain size. I've worked on plenty of teams with relatively small apps, and the difference between: 1. Cloud: "open up the cloud console and start a VM" 2. Owned hardware: "price out a server, order it, find a suitable datacenter, sign a contract, get it racked, etc." Is quite large. #1 is 15 minutes for a single team lead. #2 requires the team to agree on hardware s…

There is a large gap between "own the hardware" and "use cloud hosting". Many people rent the hardware, for example, and you can use managed databases which is one step up than "starting a vm". But your comparison isn't fair. The difference between running your own hardware and using the cloud (which is perhaps not even the relevant comparison but let's run with it) is the difference between: 1. Open up the cloud con…

I've always liked the theory of #2, I just haven't worked anywhere yet that has executed it well.

Re: Why we use our own hardware

#247

Post by mark242 » Sun, Dec 22, 2024, 5:15 PM UTC

zosima wrote: post #83

johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

Cloud expands the capabilities of what one team can manage by themselves, enabling them to avoid a huge amount of internal politics. This is worth astronomical amounts of money in big corps.

This is absolutely spot on.

What do you mean, I can't scale up because I've used my hardware capex budget for the year?

Re: Why we use our own hardware

#248

Post by dietr1ch » Sun, Dec 22, 2024, 5:18 PM UTC

toast0 wrote: post #126

dietr1ch wrote: post #100

Earlier quoted context omitted.

Does it? I've seen outages around "Sorry, us-west_carolina-3 is down". AWS is particularly good at keeping you aware of their datacenters.

It can be useful. I run a latency sensitive service with global users. A cloud lets me run it in 35 locations dealing with one company only. Most of those locations only have traffic to justify a single, smallish, instance. In the locations where there's more traffic, and we need more servers, there are more cost effective providers, but there's value in consistency. Elasticity is nice too, we doubled our instance co…

I agree it can be useful (latency, availability, using off-peak resources), but running globally should be a default and people should opt-in into fine-grained control and responsibility.

From outside it seems that either AWS picked the wrong default to present their customers, or that it's unreasonably expensive and it drives everyone into the in-depth handling to try to keep cloud costs down.

Re: Why we use our own hardware

#249

Post by sfilmeyer » Sun, Dec 22, 2024, 5:19 PM UTC

fulafel wrote: post #194

UltraSane wrote: post #151

Earlier quoted context omitted.

"All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding," This is false. AWS infrastructure is vastly more secure than almost all company data centers. AWS has a rule that the same person cannot have logical access and physical access to the same storage device. Very few companies have enough IT people to have this rule. The AWS KMS is vastly m…

OTOH: 1. big clouds are very lucrative targets for spooks, your data seem pretty likely to be hoovered up as "bycatch" (or maybe main catch depending on your luck) by various agencies and then traded around as currency 2. you never hear about security probems (incidents or exposure) in the platforms, there's no transparency 3. better than most coporate stuff is a low bar

>3. better than most corporate stuff is a low bar

I think it's a very relevant bar, though. The top level commenter made points about "a business of just about any size", which seems pretty exactly aligned with "most corporate stuff".

Re: Why we use our own hardware

#250

Post by wslh » Sun, Dec 22, 2024, 5:22 PM UTC

UltraSane wrote: post #151

johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

"All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding," This is false. AWS infrastructure is vastly more secure than almost all company data centers. AWS has a rule that the same person cannot have logical access and physical access to the same storage device. Very few companies have enough IT people to have this rule. The AWS KMS is vastly m…

From a critical perspective, your comment made me think about the risks posed by rogue IT personnel, especially at scale in the cloud. For example, Fastmail is a single point of failure as a DoS target, whereas attacking an entire datacenter can impact multiple clients simultaneously. It all comes down to understanding the attack vectors.