Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We decided to move 90% of our workload from the cloud to on-prem infrastructure

medium.com

Post reply on HN
Print view View original

181–190 of 218 posts

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#181

Post by philliphaydon »

GTP wrote: post #120
philliphaydon wrote: post #88

Earlier quoted context omitted.

> A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t. Ani. If the hardware you’re running on is dying. In ec2 you stop it and start it. It’s on new hardware. If you run bare metal you’re screwed. Disk is dying ? Don’t matter cos your data exists multiple times over in AWS elastic storage. With bare metal y…

>Disk is dying ? Don’t matter cos your data exists multiple times over in AWS elastic storage. With bare metal you got to shut down and replace. Not really, this is what RAID is for

Sure, raids some drives together, you still have downtime when you switch it off replace the drive switch it on wait for it to rebuild.

Or you can fork out the extra cost to have hot swappable.

If you're saying: "I dont want to pay $300 / month for EC2 instances at AWS when I can get the same hardware specs for $150 at X"

Chances are you're getting a shitty old desktop or barebones rack that lacks features like hot swap and raids.

Do you believe you're getting redundancy when you go with hetzner and a cheap desktop grade processor, ddr4 non ecc memory, and a consumer grade SSD?

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#182

Post by lostmsu »

dagw wrote: post #76
recuter wrote: post #3

After spending $100k in a year.. After doing the math, We decided to purchase three workstations at a total cost of $17k. One is a GPU-based workstation with two RTX 3090s and an Intel i9–12900 CPU, and another two workstations with 16 cores AMD Ryzen 5950X CPUs. It took us a few FTE days to set those up to our satisfaction with slurm, NFS, backups, and several other services. We noticed that our RTXs, although consi…

We noticed that our RTXs, although considered gaming cards, are comparable (if not better) in performance to Tesla V100 You don't buy Tesla cards because you care about maximizing operations pr second, you buy them because you care about operations pr KWh.

That's a good point. 3090 RTX may be 1.5x faster than my Titan V, but Titan V consumes 2x less energy on the same load while being much older generation card.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#183

Post by ternaryoperator »

yubiox wrote: post #130

It is on-premises not on-premise. Why is this so hard?

> Why is this so hard?

Well, for one, because it's an exceptionally rare English word that is singular in meaning but plural in its usage. In addition, there is an unrelated English word that is singular in both meaning and construction. That confusion would naturally arise seems almost pre-ordained.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#184

Post by dagw »

lostmsu wrote: post #182
dagw wrote: post #76

Earlier quoted context omitted.

We noticed that our RTXs, although considered gaming cards, are comparable (if not better) in performance to Tesla V100 You don't buy Tesla cards because you care about maximizing operations pr second, you buy them because you care about operations pr KWh.

That's a good point. 3090 RTX may be 1.5x faster than my Titan V, but Titan V consumes 2x less energy on the same load while being much older generation card.

Plus less energy consumption means less heat generated which means you can pack them tighter and you can spend less on cooling etc etc. At scale these things really add app.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#185

Post by vidarh »

treis wrote: post #71
vidarh wrote: post #70

Earlier quoted context omitted.

If you have to use more than one a day a month managing the server, you are doing something wrong. I used to manage multiple racks worth of servers on top of managing the 1k containers running on them, maintaining the (pre-kubernetes) orchestration software I had written to deploy containers to our servers, and still had time left over to spend the majority of my time on the architecture and project management of new…

I'd say if you're not spending one day a month you're doing something wrong. Namely, you're not testing your back ups and disaster recovery often enough.

That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup.

It also should not take anywhere near a day per server per month - if it does, then in a disaster scenario it means you're unable to recover at a reasonable pace.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#187

Post by whiplash451 »

greedo wrote: post #175
whiplash451 wrote: post #135

Earlier quoted context omitted.

> A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t. Definitely not true. With a dedicated server, you need to handle backup and security yourself.

With ec2 you still need to backup. You still need to validate the backups. Security it still something you need to do since an instance is still just a VM. Same with s3 buckets etc. Google for public s3 bucket "breaches." You still need to apply patches, to configure access, to expand volumes, to configure VPCs and security groups.

As far as I understand, when you use an EC2 volume, it is already backed up for you. It is not the case for an OVH dedicated server. For instance, backing up your OS image is a lot more work with a dedicated server than with an EC2 instance.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#188

Post by vidarh »

philliphaydon wrote: post #179
marcosdumay wrote: post #131

Earlier quoted context omitted.

> In ec2 you stop it and start it. It’s on new hardware. If you run bare metal you’re screwed. You mean, you reimage? That is the slow step, you reimage, and plug the new server. Wait a bit, and your service has one more server. > With bare metal you got to shut down and replace. You take the disk out and plug a new one. You don't turn things off because of a disk. No doubt, those are costly. They are also rare (disk…

> You mean, you reimage? That is the slow step, you reimage, and plug the new server. Wait a bit, and your service has one more server. No. When you /stop/ an EC2 instance, and /start/ it again, it moves. You do not need to reimage. This is even requested from AWS when they are having hardware failures and need to move customers off so they can decomission the hardware. They request you stop / start the instance, if…

> If you have a storage array sure. But if you're getting bare metal hosting from a provider, you're not always getting hot swappable storage arrays.

If you have any server-level hardware bought in the last 20 years or so, it will have the drives in hot-swappable bays. If you then choose to not set it up in RAID, it's just incompetence.

> If you have a storage array sure. But if you're getting bare metal hosting from a provider, you're not always getting hot swappable storage arrays.

If you're getting bare metal hosting from anywhere including your own colo, you have failover and the ability to order replacements while your system is still running. This is only an issue if you're architecture is fundamentally flawed, in which case you're likely to mess things up whether you're on bare metal or in a cloud.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#189

Post by vidarh »

philliphaydon wrote: post #180
ludjer wrote: post #146

Earlier quoted context omitted.

Not really If you running bare metal and have a San you can easily change what it is pointing to. Also most bare metal servers have redundancy and disks can be changed with 0 down time.

So now you're throwing a ton more money at something, costing you in purchasing the bare metal, but also the cost to maintain it. If you want to build compariable redundancy as you get with EC2 it will cost you. It wont be cheaper.

I have no idea what you think the costs of this is. I have managed setups like that. Every year we priced out what a cost to EC2 would cost us, and every year it was about 3x the cost of running our own, with my time - accounted for to the hour - of running the system added in. Every year we also priced out Hetzner and a few other options. After a years Hetzner eventually won out (colo space in Germany was cheaper than where we were in London). So we tied Hetzner servers into our private cloud layer, and migrated containers and shut down servers as it fit into our schedule. Not having to physically go to the colo's to swap drives now and again saved me an average of maybe 2 days a month to deal with several racks worth of hardware.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#190

Post by vidarh »

philliphaydon wrote: post #181
GTP wrote: post #120

Earlier quoted context omitted.

>Disk is dying ? Don’t matter cos your data exists multiple times over in AWS elastic storage. With bare metal you got to shut down and replace. Not really, this is what RAID is for

Sure, raids some drives together, you still have downtime when you switch it off replace the drive switch it on wait for it to rebuild. Or you can fork out the extra cost to have hot swappable. If you're saying: "I dont want to pay $300 / month for EC2 instances at AWS when I can get the same hardware specs for $150 at X" Chances are you're getting a shitty old desktop or barebones rack that lacks features like hot s…

There is no extra cost to have hot swappable unless you're considering buying consumer grade hardware, but consumer grade hardware is more expensive to host because it won't fit in 1U bays.

> Do you believe you're getting redundancy when you go with hetzner and a cheap desktop grade processor, ddr4 non ecc memory, and a consumer grade SSD?

Irrespective of how much I'd skimp on the hardware, I always have a HA setup, including on EC2, so it doesn't matter in any case.

Post reply on HN