Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We decided to move 90% of our workload from the cloud to on-prem infrastructure

medium.com

Post reply on HN
Print view View original

211–218 of 218 posts

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#211

Post by fxtentacle »

v-yadli wrote: post #210
fxtentacle wrote: post #200

Earlier quoted context omitted.

If I remember correctly, the IEEE 802.3an line coding overhead is around 2.6ms for each time you switch between SPF+ and RJ-45. That's in line with my measurements. SPF+ switch to OM3 fiber to SPF+ PCIe card => 0.2ms SPF+ switch to RJ-45 cable to RJ-45 PCIe card => around 3ms

NAS-[10GbE]->switch-[1GbE]->asus router, 2 hops ping test: ``` yatli@yatao-nas ~ % ping 192.168.50.1 PING 192.168.50.1 (192.168.50.1) 56(84) bytes of data. 64 bytes from 192.168.50.1: icmp_seq=1 ttl=64 time=0.236 ms 64 bytes from 192.168.50.1: icmp_seq=2 ttl=64 time=0.195 ms 64 bytes from 192.168.50.1: icmp_seq=3 ttl=64 time=0.300 ms 64 bytes from 192.168.50.1: icmp_seq=4 ttl=64 time=0.271 ms 64 bytes from 192.168.50…

OK it seems mistake was to put a RJ-45 transceiver into an SPF port, if your direct RJ-45 line is so much faster.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#212

Post by v-yadli »

fxtentacle wrote: post #211
v-yadli wrote: post #210

Earlier quoted context omitted.

NAS-[10GbE]->switch-[1GbE]->asus router, 2 hops ping test: ``` yatli@yatao-nas ~ % ping 192.168.50.1 PING 192.168.50.1 (192.168.50.1) 56(84) bytes of data. 64 bytes from 192.168.50.1: icmp_seq=1 ttl=64 time=0.236 ms 64 bytes from 192.168.50.1: icmp_seq=2 ttl=64 time=0.195 ms 64 bytes from 192.168.50.1: icmp_seq=3 ttl=64 time=0.300 ms 64 bytes from 192.168.50.1: icmp_seq=4 ttl=64 time=0.271 ms 64 bytes from 192.168.50…

OK it seems mistake was to put a RJ-45 transceiver into an SPF port, if your direct RJ-45 line is so much faster.

Yeah I think so. There might also be better converter modules. Anyway, my setup is caused by an oversight of not burying light pipes into the floor. Otherwise I guess I'll go with SFP+ too. I'm using TP-Link TL-ST1005 which is the cheapest 10GbE switch out there and it cost like $150. Also RJ-45 adapters/switches run anecdotally hotter than SFP+ ones.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#213

Post by vidarh »

stephbu wrote: post #209
vidarh wrote: post #195

Earlier quoted context omitted.

The thing is, you can buy your way out of most of these considerations by either renting colo space or renting managed servers until/unless you're at a scale where there are savings to doing it yourself. This is a commodity service with margins a tiny fraction of AWS' margins. Buying services at those levels instead of doing hosting on prem still nets you 80%-90% of the savings vs. cloud. Sometimes it nets you greate…

I think I covered those poin in calling out the cost model elements - renting the “bottom third” of the costs via managed facilities, BMaaS etc. helps in terms of reducing or eliminating capital expenses and some human toil especially in a more stable business that doesn’t benefit from per-minute lease terms or multi-year. As I called out elsewhere, sizing the hosting model to the economic model is really important.…

If human costs dominate, you're doing something very wrong, and in any case renting the services only obscure this, and you can externalise that just as easily with managed servers by outsourcing the management.

What is truly expensive is buying a bundled service with massive margins.

Sometimes you have the luxury of not caring, e.g. when building very high margin, low scale tools where human costs dominate due to dev or other parts of the business, but for anything that starts requiring hosting costs beyond even as low as a couple of k a month, you're leaving money on the table.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#214

Post by vidarh »

maccard wrote: post #206
vidarh wrote: post #189

Earlier quoted context omitted.

I have no idea what you think the costs of this is. I have managed setups like that. Every year we priced out what a cost to EC2 would cost us, and every year it was about 3x the cost of running our own, with my time - accounted for to the hour - of running the system added in. Every year we also priced out Hetzner and a few other options. After a years Hetzner eventually won out (colo space in Germany was cheaper th…

> Every year we priced out what a cost to EC2 would cost us, and every year it was about 3x the cost of running our own > we tied Hetzner servers into our private cloud layer, and migrated containers and shut down servers as it fit Building your own services on top of AWS is always going to come out more expensive. EC2 + EBS volumes alone are going to be more expensive than going with hetzner (particularly if you're…

The cost of building out our private setup was my time for about a month.

At the time we didn't have a choice, as nothing like Fargate existed, but today it's also easier to do setups like the one we did. It mostly involved rsyncing base images over, rsync and a super simple storage service for backups, a LDAP based directory service, and and a thin layer over vzctl (first) and docker when that became an option, coupled with a VPN setup to tie our locations together, and a reverse proxy setup that did dynamic lookups in our private DNS fed from LDAP.

It is hard to do as a multitenant public service, it's trivially easy to do as an internal tool that needs to support only exactly what you need.

I've built out setups like this for a number of clients since, and it's typically 1-3 months of work to automate pretty much everything depending on complexity, and so it pays for itself quickly from a very low scale.

The first company I did this at wouldn't have been profitable at all if we'd relied on AWS

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#215

Post by vidarh »

treis wrote: post #208
vidarh wrote: post #205

Earlier quoted context omitted.

Most decent bare metal setups abstracts away the same class of "your hardware died" problems. E.g. my first rule on this is everything runs in containers whether I run in a cloud environment or on bare metal. The OS image is identical, and include tying into service discovery and a suitable orchestration mechanism (which can range from something trivially simple to, say, Kubernetes). Any modern server hardware has IP…

Containers running some work load dying is a lot different from a HDD going kaput on your production DB server. Or someone compromising your file system and encrypting all of your user uploads. I don't think anyone is claiming it's a particularly hard problem. It's the opposite. We're talking about 1 hour a month. Even that little effort still ~$200 a month and a $200 a month managed DB instance is pretty beefy.

None of this is any different I a cloud setup. You still need to check the integrity of your backups. You still need to have appropriate failover. The effort is pretty much the same. In a competent setup it's automated, so you're not spending an hour a month per server. Not even for the production DB server, because you have replicas and setups to automate the switch, and setups to bring back snapshots from backups.

Been there, done that, many times, and I know exactly what it costs people, because I billed by the hour for it, and your cost assumptions are just way off.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#217

Post by hansel_der »

fxtentacle wrote: post #211
v-yadli wrote: post #210

Earlier quoted context omitted.

NAS-[10GbE]->switch-[1GbE]->asus router, 2 hops ping test: ``` yatli@yatao-nas ~ % ping 192.168.50.1 PING 192.168.50.1 (192.168.50.1) 56(84) bytes of data. 64 bytes from 192.168.50.1: icmp_seq=1 ttl=64 time=0.236 ms 64 bytes from 192.168.50.1: icmp_seq=2 ttl=64 time=0.195 ms 64 bytes from 192.168.50.1: icmp_seq=3 ttl=64 time=0.300 ms 64 bytes from 192.168.50.1: icmp_seq=4 ttl=64 time=0.271 ms 64 bytes from 192.168.50…

OK it seems mistake was to put a RJ-45 transceiver into an SPF port, if your direct RJ-45 line is so much faster.

either that, or the equipment is in awe because you mixed up its acronym

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#218

Post by robertlagrant »

GordonS wrote: post #197
simonebrunozzi wrote: post #148

Earlier quoted context omitted.

> In addition: long before AWS you could easily rent virtualized or dedicated servers. I disagree with this statement. Yes, you could rent, but not by the hour and based on compute power, and couldn't rent extra storage again by the hour and by the GB. Plus, you couldn't interact with these "virtual servers" through APIs. I was at AWS 2008-2014 (early days!), and I think you should consider the impact of the "on-dema…

> Yes, you could rent, but not by the hour and based on compute power, and couldn't rent extra storage again by the hour and by the GB Right - because it's so cheap you don't need to rent by the hour.

Think of the case we might have at my company: we occasionally need a load of GPUs to train a new ML model. We spin up some beefy VMs with stonking great GPUs, churn away for a while, and then spin them down with our new ML model ready to go.
Post reply on HN