Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use our own hardware

fastmail.com

Post reply on HN
Print view View original

171–180 of 547 posts

Re: Why we use our own hardware

#171

Post by sgarland »

TacticalCoder wrote: post #104
johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

> All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding ... And moreover most of the actual interesting things, like having VM templates and stateless containers, orchestration, etc. is very easy to run yourself and gets you 99.9% of the benefits of the cloud. About just any and every service is available as container file already written for…

> If an individual can do that, a company, no matter its size, can do it too.

Fully agreed. I don't have physical HA – if someone stole my rack, I would be SOL – but I can easily ride out a power outage for as long as I want to be hauling cans of gasoline to my house. The rack's UPS can keep it up at full load for at least 30 minutes, and I can get my generator running and hooked up in under 10. I've done it multiple times. I can lose a single server without issue. My only SPOF is internet, and that's only by choice, since I can get both AT&T and Spectrum here, and my router supports dual-WAN with auto-failover.

> And arguably 99.9% of all the companies out there don't have the need for an infra as powerful as the one most homelab enthusiast have.

THIS. So many people have no idea how tremendously fast computers are, and how much of an impact latency has on speed. I've benchmarked my 12-year old Dells against the newest and shiniest RDS and Aurora instances on both MySQL and Postgres, and the only ones that kept up were the ones with local NVMe disks. Mine don't even technically have _local_ disks; they're NVMe via Ceph over Infiniband.

Does that scale? Of course not; as soon as you want geo-redundant, consistent writes, you _will_ have additional latency. But most smaller and medium companies don't _need_ that.

Re: Why we use our own hardware

#173

Post by noprocrasted »

briHass wrote: post #86

The biggest win with running your own infra is disk/IO speeds, as noted here and in DHH's series on leaving cloud ( https://world.hey.com/dhh/we-have-left-the-cloud-251760fb ) The cloud providers really kill you on IO for your VMs. Even if 'remote' SSDs are available with configurable ($$) IOPs/bandwidth limits, the size of your VM usually dictates a pitiful max IO/BW limit. In Azure, something like a 4-core 16GB RAM…

Biggest issue isn't even sequential speed but latency. In the cloud all persistent storage is networked and has significantly more latency than direct-attached disks. This is a physical (speed of light) limit, you can't pay your way out of it, or throw more CPU at it. This has a huge impact for certain workloads like relational databases.

Re: Why we use our own hardware

#174

Post by sanderjd »

johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

Also, by the way, I found it interesting that you framed your side of this disagreement as the technically correct one, but then included this:

> a desire to not centralize the Internet

This is an ideological stance! I happen to share this desire. But you should be aware of your own non-technical - "emotional" - biases when dismissing the arguments of others on the grounds that they are "emotional" and+l "fanatical".

Re: Why we use our own hardware

#175

Post by maccard »

tylerchurch wrote: post #123
sgarland wrote: post #103

Earlier quoted context omitted.

I’m not convinced this is entirely true. The upfront cost if you don’t have the skills, sure – it takes time to learn Linux administration, not to mention management tooling like Ansible, Puppet, etc. But once those are set up, how is it different? AWS is quite clear with their responsibility model that you still have to tune your DB, for example. And for the setup, just as there are Terraform modules to do everythin…

I think this is only true for teams and apps of a certain size. I've worked on plenty of teams with relatively small apps, and the difference between: 1. Cloud: "open up the cloud console and start a VM" 2. Owned hardware: "price out a server, order it, find a suitable datacenter, sign a contract, get it racked, etc." Is quite large. #1 is 15 minutes for a single team lead. #2 requires the team to agree on hardware s…

I work for a 50 person subsidiary of a 30k person organisation. I needed a domain name. I put in the purchase request and 6 months later eventually gave up, bought it myself and expensed it.

Our AWS account is managed by an SRE team. It’s a 3 day turnaround process to get any resources provisioned, and if you don’t get the exact spec right (you forgot to specify the iops on the volume? Oops) 3 day turnaround. Already started work when you request an adjustment? Better hope as part of your initial request you specified backups correctly or you’re starting again.

The overhead is absolutely enormous, and I actually don’t even have billing access to the AWS account that I’m responsible for.

Re: Why we use our own hardware

#176

Post by krsgjerahj »

floating-io wrote: post #114
jeffbee wrote: post #75

Earlier quoted context omitted.

The problem with your claims here is they can only be right if the entire industry is experiencing mass psychosis. I reject a theory that requires that, because my ego just isn't that large. I once worked for several years at a publicly traded firm well-known for their return-to-on-prem stance, and honestly it was a complete disaster. The first-party hardware designs didn't work right because they didn't have the har…

"Vendor problems" is a red herring, IMO; you can have those in the cloud, too. It's been my experience that those who can build good, reliable, high-quality systems, can do so either in the cloud or on-prem, generally with equal ability. It's just another platform to such people, and they will use it appropriately and as needed. Those who can only make it work in the cloud are either building very simple systems (whi…

you forgot cogs

it's all about painting the right picture for your investors, so you make up shit and classify as cogs or opex depending on what is most beneficial for you in the moment

Re: Why we use our own hardware

#177

Post by noprocrasted »

zbentley wrote: post #163
noprocrasted wrote: post #150

Earlier quoted context omitted.

3. "Dedicated server" at any hosting provider Open their management console, press order now, 15 mins later get your server's IP address.

For purposes of this discussion, isn't AWS just a very large hosting provider? I.e. most hosting providers give you the option for virtual or dedicated hardware. So does Amazon (metal instances). Like, "cloud" was always an ill-defined term, but in the case of "how do I provision full servers" I think there's no qualitative difference between Amazon and other hosting providers. Quantitative, sure.

> Amazon (metal instances)

But you still get nickel & dimed and pay insane costs, including on bandwidth (which is free in most conventional hosting providers, and overages are 90x cheaper than AWS' costs).

Re: Why we use our own hardware

#178

Post by citrin_ru »

necovek wrote: post #148
johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

But isn't using Fastmail akin to using a cloud provider (managed email vs managed everything else)? They are similarly a service provider, and as a customer, you don't really care "who their ISP is?" The discussion matters when we are talking about building things: whether you self-host or use managed services is a set of interesting trade-offs.

Yes, FastMail is a SAAS. But there adepts of a religion which would tell you that companies like FastMail should be built on top of AWS and it is the only true way. It is good to have some counter narrative to this.

Re: Why we use our own hardware

#179

Post by liontwist »

glitchc wrote: post #87
johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

Cloud solves one problem quite well: Geographic redundancy. It's extremely costly with on-prem.

Cloud noob here. But if I have a central database what can I distribute across geographic regions? Static assets? Maybe a cache?

Re: Why we use our own hardware

#180

Post by noprocrasted »

rtfusgihkuj wrote: post #170
noprocrasted wrote: post #149

Earlier quoted context omitted.

> identity management (AAD/IAM) Do you mean for administrative access to the machines (over SSH, etc) or for "normal" access to the hosted applications? Admin access: Ansible-managed set of UNIX users & associated SSH public keys, combined with remote logging so every access is audited and a malicious operator wiping the machine can't cover their tracks will generally get you pretty far. Beyond that, there are commer…

No, using Ansible to distribute public keys does not get you very far. It's fine for a personal project or even a team of 5-6 with a handful, but beyond that you really need a better way to onboard, offboard, and modify accounts. If you're doing anything but a toy project, you're better off starting off with something like IPA for host access controls.

What's the risk you're trying to protect against, that a "better" (which one?) way would mitigate that this one wouldn't?

> IPA

Do you mean https://en.wikipedia.org/wiki/FreeIPA ? That seems like a huge amalgamation of complexity in a non-memory-safe language that I feel like would introduce a much bigger security liability than the problem it's trying to solve.

I'd rather pony up the money and use Teleport at that point.

Post reply on HN