Many companies that do this look at MinIO for object storage. Given they run in AWS, GCP and Azure, they will minimize or eliminate your application rewrites. They are cloud-native by design and very fast.
We decided to move 90% of our workload from the cloud to on-prem infrastructure
201–210 of 218 posts
Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure
#202Earlier quoted context omitted.
I'd say if you're not spending one day a month you're doing something wrong. Namely, you're not testing your back ups and disaster recovery often enough.
That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup. It also should not take anywhere near a day per server per month - if it does, then in a disaster scenario it means you're unable to recover at a reasonable pace.
>That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup.
It is additional time because the cloud handles the whole class of "your hardware died" problems.
Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure
#203Earlier quoted context omitted.
As far as I understand, when you use an EC2 volume, it is already backed up for you. It is not the case for an OVH dedicated server. For instance, backing up your OS image is a lot more work with a dedicated server than with an EC2 instance.
That addresses one possible set of failure scenarios. If you think that means you have a backup that is guaranteed to be accessible to you whenever you need it , you don't have a backup. If your only backup is in the same cloud provider where your primary system is, you don't have a backup, you only think you do. Any reasonable dedicated setup will involve imaging your server, and so the OS image is not something you…
That's still a lot better than what you have with a dedicated server.
For you to lose your OS image with a dedicated server only takes your HDD to die.
For you to lose your OS image on EC2 (where you made a snapshot of your volume in one-click) would take a lot of shitstorm to happen at AWS -- as I presume that they backup across sites.
Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure
#204Earlier quoted context omitted.
That addresses one possible set of failure scenarios. If you think that means you have a backup that is guaranteed to be accessible to you whenever you need it , you don't have a backup. If your only backup is in the same cloud provider where your primary system is, you don't have a backup, you only think you do. Any reasonable dedicated setup will involve imaging your server, and so the OS image is not something you…
That's still a lot better than what you have with a dedicated server. For you to lose your OS image with a dedicated server only takes your HDD to die. For you to lose your OS image on EC2 (where you made a snapshot of your volume in one-click) would take a lot of shitstorm to happen at AWS -- as I presume that they backup across sites.
> For you to lose your OS image with a dedicated server only takes your HDD to die.
Only if you don't have a backup.
Why in the world do you think anyone would store their only copy of an OS image on a single server?
For systems I set up, to start with, the OS is mostly immutable, booted and updated transparently to match a master image. If it gets destroyed, we just image a new server. The applications all run in containers, based on images stored on replicated file servers. If they get destroyed, we just re-deploy on a different server (in fact, automatically redeploying is trivial).
Only the application data is unique to running servers, and that needs to be backed up just as much whether those containers run in a cloud environment or locally, and again it's trivial to have automation in place for the backup and re-deployment of that. Been there, done that many times.
> For you to lose your OS image on EC2 (where you made a snapshot of your volume in one-click) would take a lot of shitstorm to happen at AWS -- as I presume that they backup across sites.
For me to lose my data on any bare metal system I've run, multiple servers in at least two different data centres operated by at different companies would need to fail at the same time. This is not hard to set up, and it's a one of setup. You then need to test your backups, just as you need to with AWS - an untested backup is not a backup.
But your assumptions of failure scenarios is also flawed. You need to protect against e.g. disgruntled employees, hackers, bugs as well. If you rely on the same security to protect your backups as your main setup, you don't have a backup.
EC2 is great when you can justify the cost, but it does not remove the need for a proper backup policy and processes to test them.
Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure
#205Earlier quoted context omitted.
That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup. It also should not take anywhere near a day per server per month - if it does, then in a disaster scenario it means you're unable to recover at a reasonable pace.
>That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup. It is additional time because the cloud handles the whole class of "your hardware died" problems.
Most decent bare metal setups abstracts away the same class of "your hardware died" problems. E.g. my first rule on this is everything runs in containers whether I run in a cloud environment or on bare metal. The OS image is identical, and include tying into service discovery and a suitable orchestration mechanism (which can range from something trivially simple to, say, Kubernetes). Any modern server hardware has IPMI or something equivalent, which means you plug it in, configure the IPMI, configure network boot from a tftp/bootp or similar server holding your installation image, and from there on out you're deploying containers the same way as you would in a cloud environment, and back them up and arrange for failover the same way as in a cloud environment.
I've set up more than one hybrid setup where you didn't need to know if your workload was running in AWS or Hetzner or somewhere else, so we could use AWS for elasticity and Hetzner to keep cost down.
This is not a hard problem. And if people don't have the right skills in house, it's easy to outsource this (I for one used to make my living of automating setups like this and operating them on a retainer basis).
Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure
#206Earlier quoted context omitted.
So now you're throwing a ton more money at something, costing you in purchasing the bare metal, but also the cost to maintain it. If you want to build compariable redundancy as you get with EC2 it will cost you. It wont be cheaper.
I have no idea what you think the costs of this is. I have managed setups like that. Every year we priced out what a cost to EC2 would cost us, and every year it was about 3x the cost of running our own, with my time - accounted for to the hour - of running the system added in. Every year we also priced out Hetzner and a few other options. After a years Hetzner eventually won out (colo space in Germany was cheaper th…
> Every year we priced out what a cost to EC2 would cost us, and every year it was about 3x the cost of running our own
> we tied Hetzner servers into our private cloud layer, and migrated containers and shut down servers as it fit
Building your own services on top of AWS is always going to come out more expensive. EC2 + EBS volumes alone are going to be more expensive than going with hetzner (particularly if you're not looking at reserved instances, and not utilising spot for burst). You mentioned that you are building your own private cloud layer and migrated containers; the cost of building that out in the first place is likely enormous compared to building and running on top of fargate.
Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure
#207Earlier quoted context omitted.
> Especially if you need multiple hops, fiber is just faster. In my tests, 0.2ms vs 3ms roundtrip time for 10G SPF vs. 10G RJ-45 ime if your unloaded latency on copper ethernet exceeds 0.2ms per hop there is some form of powersaving involved
If I remember correctly, the IEEE 802.3an line coding overhead is around 2.6ms for each time you switch between SPF+ and RJ-45. That's in line with my measurements. SPF+ switch to OM3 fiber to SPF+ PCIe card => 0.2ms SPF+ switch to RJ-45 cable to RJ-45 PCIe card => around 3ms
pretty wild, considering my phone averages at 2.2ms ROUND trip time when pinging the router.
Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure
#208Earlier quoted context omitted.
>That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup. It is additional time because the cloud handles the whole class of "your hardware died" problems.
Most decent bare metal setups abstracts away the same class of "your hardware died" problems. E.g. my first rule on this is everything runs in containers whether I run in a cloud environment or on bare metal. The OS image is identical, and include tying into service discovery and a suitable orchestration mechanism (which can range from something trivially simple to, say, Kubernetes). Any modern server hardware has IP…
Containers running some work load dying is a lot different from a HDD going kaput on your production DB server. Or someone compromising your file system and encrypting all of your user uploads.
I don't think anyone is claiming it's a particularly hard problem. It's the opposite. We're talking about 1 hour a month. Even that little effort still ~$200 a month and a $200 a month managed DB instance is pretty beefy.
Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure
#209Earlier quoted context omitted.
It’s pretty hard to generalize this without qualifiers. Electricity can be the most expensive problem, but it requires carefully planned control of the other factors - e.g large scale with highly automated servers, network, and meat-reducing control planes to become true. Otherwise factors such as people and under-utilization can especially dominate smaller and/or less efficient facilities. Thinking thru the factors,…
The thing is, you can buy your way out of most of these considerations by either renting colo space or renting managed servers until/unless you're at a scale where there are savings to doing it yourself. This is a commodity service with margins a tiny fraction of AWS' margins. Buying services at those levels instead of doing hosting on prem still nets you 80%-90% of the savings vs. cloud. Sometimes it nets you greate…
I think I covered those poin in calling out the cost model elements - renting the “bottom third” of the costs via managed facilities, BMaaS etc. helps in terms of reducing or eliminating capital expenses and some human toil especially in a more stable business that doesn’t benefit from per-minute lease terms or multi-year. As I called out elsewhere, sizing the hosting model to the economic model is really important.
Even in the pure rental or managed BMaaS, the human cost can quickly dominate the economic model. Owning machines and OS’s is expensive at anything more than a couple of racks of machines. Eliminating people and human change/release from touching things in the datacenter is probably the first priority. Otherwise it is hard to consistently drive that human number down and meet service quality expectations for 24x7.
Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure
#210Earlier quoted context omitted.
> Especially if you need multiple hops, fiber is just faster. In my tests, 0.2ms vs 3ms roundtrip time for 10G SPF vs. 10G RJ-45 ime if your unloaded latency on copper ethernet exceeds 0.2ms per hop there is some form of powersaving involved
If I remember correctly, the IEEE 802.3an line coding overhead is around 2.6ms for each time you switch between SPF+ and RJ-45. That's in line with my measurements. SPF+ switch to OM3 fiber to SPF+ PCIe card => 0.2ms SPF+ switch to RJ-45 cable to RJ-45 PCIe card => around 3ms
NAS-[10GbE]->switch-[1GbE]->asus router, 2 hops ping test:
```
yatli@yatao-nas ~ % ping 192.168.50.1
PING 192.168.50.1 (192.168.50.1) 56(84) bytes of data.
64 bytes from 192.168.50.1: icmp_seq=1 ttl=64 time=0.236 ms
64 bytes from 192.168.50.1: icmp_seq=2 ttl=64 time=0.195 ms
64 bytes from 192.168.50.1: icmp_seq=3 ttl=64 time=0.300 ms
64 bytes from 192.168.50.1: icmp_seq=4 ttl=64 time=0.271 ms
64 bytes from 192.168.50.1: icmp_seq=5 ttl=64 time=0.153 ms
64 bytes from 192.168.50.1: icmp_seq=6 ttl=64 time=0.216 ms
64 bytes from 192.168.50.1: icmp_seq=7 ttl=64 time=0.300 ms
64 bytes from 192.168.50.1: icmp_seq=8 ttl=64 time=0.165 ms
64 bytes from 192.168.50.1: icmp_seq=9 ttl=64 time=0.163 ms
64 bytes from 192.168.50.1: icmp_seq=10 ttl=64 time=0.265 ms
64 bytes from 192.168.50.1: icmp_seq=11 ttl=64 time=0.174 ms
64 bytes from 192.168.50.1: icmp_seq=12 ttl=64 time=0.272 ms
64 bytes from 192.168.50.1: icmp_seq=13 ttl=64 time=0.397 ms
64 bytes from 192.168.50.1: icmp_seq=14 ttl=64 time=0.256 ms
^C
--- 192.168.50.1 ping statistics ---
14 packets transmitted, 14 received, 0% packet loss, time 13168ms
rtt min/avg/max/mdev = 0.153/0.240/0.397/0.065 ms
```