We decided to move 90% of our workload from the cloud to on-prem infrastructure • Hacker News Board

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#131

Post by marcosdumay » Tue, Apr 12, 2022, 4:22 PM UTC

philliphaydon wrote: post #88

tiew9Vii wrote: post #74

Earlier quoted context omitted.

I hear that said a lot. I don’t think it stands. A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t. The cloud actually requires more investment for large organisations. Previously you might of only had a handful of sysadmins but now you have a large dedicated platform team doing devops type work building…

> A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t. Ani. If the hardware you’re running on is dying. In ec2 you stop it and start it. It’s on new hardware. If you run bare metal you’re screwed. Disk is dying ? Don’t matter cos your data exists multiple times over in AWS elastic storage. With bare metal y…

> In ec2 you stop it and start it. It’s on new hardware. If you run bare metal you’re screwed.

You mean, you reimage? That is the slow step, you reimage, and plug the new server. Wait a bit, and your service has one more server.

> With bare metal you got to shut down and replace.

You take the disk out and plug a new one. You don't turn things off because of a disk.

No doubt, those are costly. They are also rare (disk failure is less rare, but still rare).

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#132

Post by whiplash451 » Tue, Apr 12, 2022, 4:23 PM UTC

the_duke wrote: post #59

manigandham wrote: post #9

> "Starting a web-based or SaaS (Software as a Service) business was virtually unheard of before the age of IaaS" Nonsense. There were plenty of SaaS startups. There was even a little event called the dotcom boom all about internet companies. This lack of history and experience is why new companies get into this cloud-first mess in the first place. Cloud is primarily for flexibility in iteration, dynamic scaling, or…

In addition: long before AWS you could easily rent virtualized or dedicated servers. It took longer to provision than EC2 and all you had for storage is a fixed amount of disk, but that was absolutely sufficient for many businesses.

OVH has a great dedicated server offering. Definitely a great bang for the buck compared to AWS (if you can handle the downsides of course: security, backup, setup, handled by yourself).

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#133

Post by whiplash451 » Tue, Apr 12, 2022, 4:25 PM UTC

maccard wrote: post #43

bsenftner wrote: post #22

Earlier quoted context omitted.

Yes, they are cheap. Running one's own server is also easy peasy; far too many think it is difficult, it is not. The most expensive part is the electricity.

> The most expensive part is the electricity. No, the most expensive part is the persons time for managing it. I can rent a monstrous Dedicated Server for $400/month from OVH, but even with a UK salary, if I have to spend more than 1 day a month on it in any shape or form (and that includes the initial setup), it's cheaper to use "the cloud" or some form of a managed service.

I think that depends a lot on which stage your startup is at, what your level of funding is, etc.

Definitely not as clear-cut as you seem to imply.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#134

Post by marcosdumay » Tue, Apr 12, 2022, 4:25 PM UTC

mschuster91 wrote: post #92

manigandham wrote: post #9

> "Starting a web-based or SaaS (Software as a Service) business was virtually unheard of before the age of IaaS" Nonsense. There were plenty of SaaS startups. There was even a little event called the dotcom boom all about internet companies. This lack of history and experience is why new companies get into this cloud-first mess in the first place. Cloud is primarily for flexibility in iteration, dynamic scaling, or…

> Companies also vastly overestimate their scale when their entire business could probably fit on a single commodity server. Indeed, but that's completely leaving out the single most important thing: backups. With all of the major clouds, snapshots are easy to do both at a VM level and data level (e.g. RDS), and the cloud provider takes care that the backups are sufficiently spread to be disaster tolerant. In contras…

> and the cloud provider takes care that the backups are sufficiently spread to be disaster tolerant

Except against the disaster of the cloud deciding it's not worth to keep you as a customer, or the cloud having a distributed failure, or the cloud getting out of business...

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#135

Post by whiplash451 » Tue, Apr 12, 2022, 4:26 PM UTC

tiew9Vii wrote: post #74

maccard wrote: post #43

Earlier quoted context omitted.

> The most expensive part is the electricity. No, the most expensive part is the persons time for managing it. I can rent a monstrous Dedicated Server for $400/month from OVH, but even with a UK salary, if I have to spend more than 1 day a month on it in any shape or form (and that includes the initial setup), it's cheaper to use "the cloud" or some form of a managed service.

I hear that said a lot. I don’t think it stands. A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t. The cloud actually requires more investment for large organisations. Previously you might of only had a handful of sysadmins but now you have a large dedicated platform team doing devops type work building…

> A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t.

Definitely not true. With a dedicated server, you need to handle backup and security yourself.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#136

Post by stephbu » Tue, Apr 12, 2022, 4:26 PM UTC

marcosdumay wrote: post #126

stephbu wrote: post #63

Earlier quoted context omitted.

To add to this statement - don’t make any investment unless you have a solid plan to fully utilize it for the full length of the purchase terms. This applies as much to AWS @ minute terms, as it does to a rental @ daily terms, as it does to co-lo capital investment @ yearly terms.

> unless you have a solid plan to fully utilize it for the full length of the purchase Hum... I'd say it's much more reasonable to look at the ROI. Making investments to supply peak demand or to hedge against rare risks is perfectly ok.

Fair point, I guess the point is that unallocated resources - space/power/servers etc. can become huge stealth money sinks, eating budget every hour of every day. Being cognizant of the consumption economics before you stump up for resources is important, as is fitting the investment model to match those economics. Setting utilization/allocation targets are just one way of measuring if those models efficiently match. This is true for any service or resource consumed.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#137

Post by BryanRamirez » Tue, Apr 12, 2022, 4:27 PM UTC

dhzhzjsbevs wrote: post #105

philliphaydon wrote: post #88

Earlier quoted context omitted.

> A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t. Ani. If the hardware you’re running on is dying. In ec2 you stop it and start it. It’s on new hardware. If you run bare metal you’re screwed. Disk is dying ? Don’t matter cos your data exists multiple times over in AWS elastic storage. With bare metal y…

The cost isn't gone. It's just included. Fact is, at scale, it's still cheaper to do it yourself. You just need to reach the scale it's worth paying people to do the managing.

I agree, with the caveat that at scale it's cheaper to do it well yourself. If that weren't true, then Amazon would not be making money on EC2.

If you don't have sysadmins and network admins with the right experience, you can easily find yourself in a bad spot with single points of failure, servers that can't be easily replaced, oversubscribed PDUs, misconfigured switches/routers... and any number of other problems that aren't occurring to me right now.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#138

Post by stephen_g » Tue, Apr 12, 2022, 4:27 PM UTC

As a random aside, I’m glad to see ‘on-prem’ emerge as a common shorthand for this, only because it always grates me to see people make the extremely common mistake of saying ‘on-premise’. Premise, of course, only ever means “an idea or theory on which a statement or action is based”, whereas the actual term is premises (“the land and buildings owned by someone, especially by a company or organisation”, as in “The security guards escorted the protesters off the premises”).

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#139

Post by AtlasBarfed » Tue, Apr 12, 2022, 4:32 PM UTC

Why did openstack fail? Or did it, was it just not adopted?

I think there is still a lot of potential for open source management of core EC2/S3/networking capabilities (aka "core AWS IAAS service"). We have a fair number of cloud abstraction layers now, and obviously kubernetes, you'd think we could as an industry produce core apis for doing resource listing, availability, etc.

Maybe some of the problem is that devs have a LOT of experience with the "ask" side of IaaS: gimme storage, gimme vms, set. But they have no experience with the "provide" side, and the sort of one-off manual nature of installing networking and machines doesn't have good standardization for "reporting available resources".

At this point, aws apis are somewhat stable. (I would bitch about the error codes and documentation... but anyway). It's obviously "good enough" after 10-15 years of them.

Are there projects that try to marry an aws-ish api, which really is a reporting and request api, with a "available resources" reporting api? Are some of these things out there?

AWS ten years ago was liberating. It was progress. It was a good thing. But Amazon is not a "do no evil" corporation, much the opposite. And you see this in AWS with its treatment of startups, open source projects, and other manipulations. They are a monopoly now, or at a minimum a dangerous cartel.

A real open source alternative would be a good thing. It would be good for the rest of FAANG, it would encourage competition by allowing lesser clouds to offer core competencies that are drop-in.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#140

Post by secabeen » Tue, Apr 12, 2022, 4:41 PM UTC

briffle wrote: post #94

bsenftner wrote: post #22

Earlier quoted context omitted.

Yes, they are cheap. Running one's own server is also easy peasy; far too many think it is difficult, it is not. The most expensive part is the electricity.

Been a sysadmin for many years. Your right, for a few computers. Once you start getting into more than a quarter rack, you also need to start worrying about cooling (which also is lots of electricity) and usually things like ensuring the electricity stays on. (UPS, generator, etc). Don't forget to monitor and service all this stuff regularly. Once your past a few racks of equipment, you have generator tests and servi…

> Once your past a few racks of equipment, you have generator tests and service appointments, redundant AC, Redundant UPS, dual power to each rack, etc. Dual Internet connection, and a link to your other server room that you use for DR, etc. The costs and complexity quickly escalate after a server or two.

This is not always the case. For computationally-focused workloads like the OP describes, without direct customer interaction, it may be reasonable to accept risk of downtime in the event of failure. If you are doing computations that take weeks to complete, and you checkpoint regularly, does it really matter if your computation finishes on Saturday morning or Monday morning? If not, you can probably accept 6 hours of downtime once every couple of years and eliminate all of the redundancy overhead described above.

In HPC, the general rule of thumb is buy your hardware if you can be sure you'll run compute on it more than ~40% of the time.