Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We decided to move 90% of our workload from the cloud to on-prem infrastructure

medium.com

Post reply on HN
Print view View original

91–100 of 218 posts

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#91

Post by [deleted] »

Tsarbomb wrote: post #78
SpaceMartini wrote: post #69

I work in HPC for a cloud provider, and fully endorse this move. Anonymously, of course. You can make an economic argument for or against cloud in practically every IT domain, but in HPC the case for on-prem is really compelling; none of the cloud networking/resiliency value-add is relevant to batch workflows, and costs per core-hour are only remotely comparable if you use spot - which is itself a major compromise. T…

If you can setup HPC at scale you can setup Ceph for object storage. It will save you so much money in the long run.

[deleted]

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#92

Post by mschuster91 »

manigandham wrote: post #9

> "Starting a web-based or SaaS (Software as a Service) business was virtually unheard of before the age of IaaS" Nonsense. There were plenty of SaaS startups. There was even a little event called the dotcom boom all about internet companies. This lack of history and experience is why new companies get into this cloud-first mess in the first place. Cloud is primarily for flexibility in iteration, dynamic scaling, or…

> Companies also vastly overestimate their scale when their entire business could probably fit on a single commodity server.

Indeed, but that's completely leaving out the single most important thing: backups. With all of the major clouds, snapshots are easy to do both at a VM level and data level (e.g. RDS), and the cloud provider takes care that the backups are sufficiently spread to be disaster tolerant.

In contrast, when you co-locate you have to take care of backups completely on your own, and with many hosters you can't even influence in which of their DCs your servers will be.

If anything, OVHs SBG fire incident should have shown everyone how hard it is to build resilient systems.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#93

Post by maccard »

vidarh wrote: post #70
maccard wrote: post #43

Earlier quoted context omitted.

> The most expensive part is the electricity. No, the most expensive part is the persons time for managing it. I can rent a monstrous Dedicated Server for $400/month from OVH, but even with a UK salary, if I have to spend more than 1 day a month on it in any shape or form (and that includes the initial setup), it's cheaper to use "the cloud" or some form of a managed service.

If you have to use more than one a day a month managing the server, you are doing something wrong. I used to manage multiple racks worth of servers on top of managing the 1k containers running on them, maintaining the (pre-kubernetes) orchestration software I had written to deploy containers to our servers, and still had time left over to spend the majority of my time on the architecture and project management of new…

Broadly speaking, managing 100 "cloud" servers is roughly the same amount of work as managing 2. Moving from managing 2 cloud servers to 100 is trivial. Moving from 1 server to 2 is an architecture problem. That architecture problem sometimes comes up when it comes to scaling too. it's the cattle vs pets problem. The last thing you want to find out is that someone ssh'ed in and installed a package that's required for your service to run, _during_ vertical resizing, and that sort of thing is far more common in systems with 1 server instead of many. If you're running containers, you would likely have even less overhead (and save money) by running on ECS/DigitalOcean/Azure App Service, and if you're running a big old monolith, you likely need > 1 instance for some sort of redundancy anyway.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#94

Post by briffle »

bsenftner wrote: post #22
xuki wrote: post #14

Earlier quoted context omitted.

I'd say don't even consider colo unless you have a specific use case. Rented dedicated servers are cheap, let someone else take care of the hardware.

Yes, they are cheap. Running one's own server is also easy peasy; far too many think it is difficult, it is not. The most expensive part is the electricity.

Been a sysadmin for many years. Your right, for a few computers. Once you start getting into more than a quarter rack, you also need to start worrying about cooling (which also is lots of electricity) and usually things like ensuring the electricity stays on. (UPS, generator, etc). Don't forget to monitor and service all this stuff regularly.

Once your past a few racks of equipment, you have generator tests and service appointments, redundant AC, Redundant UPS, dual power to each rack, etc. Dual Internet connection, and a link to your other server room that you use for DR, etc. The costs and complexity quickly escalate after a server or two.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#95

Post by zozbot234 »

SpaceMartini wrote: post #69

I work in HPC for a cloud provider, and fully endorse this move. Anonymously, of course. You can make an economic argument for or against cloud in practically every IT domain, but in HPC the case for on-prem is really compelling; none of the cloud networking/resiliency value-add is relevant to batch workflows, and costs per core-hour are only remotely comparable if you use spot - which is itself a major compromise. T…

Scalability might still be a major advantage if you have infrequent enough needs for HPC-scale compute. Basically, use the cloud service as a pure computational "grid". But the pitfalls of cloud business models (including sky-high costs for data egress) often make that unworkable.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#96

Post by briffle »

SpaceMartini wrote: post #73
SpaceMartini wrote: post #69

I work in HPC for a cloud provider, and fully endorse this move. Anonymously, of course. You can make an economic argument for or against cloud in practically every IT domain, but in HPC the case for on-prem is really compelling; none of the cloud networking/resiliency value-add is relevant to batch workflows, and costs per core-hour are only remotely comparable if you use spot - which is itself a major compromise. T…

As an addendum to this: if you absolutely must use cloud, stick with AWS. Using Azure is (IMO) a fucking miserable experience and their only advantage (InfiniBand) is better served by buying your own hardware. GCP and OCI might be fine if you are getting a lot of credits, but the skills will not be useful down the line - while AWS is expensive, you will at least learn a bunch of in-demand operational skills.

So nobody got fired for buying IBM?

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#97

Post by dgb23 »

nikanj wrote: post #54

1) You save a metric boatload of money this way 2) Onprem experience looks bad on your resume, compared to cloud experience 3) Incentives work for people, and resume-driven development is key, as our industry is very stingy in passing the savings from 1) to the developers

Many care more about cs and engineering fundamentals rather than bespoke products, even more people only care about having their problem solved and how much it costs.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#98

Post by jdrc »

sokoloff wrote: post #20
lioeters wrote: post #13

Just want to highlight how futuristic the author's title is: "Computational Biologist, Head of Protein Design @ Enzymit". Why they moved to on-prem: lower and more predictable cost. At a public cloud provider, they lost thousands of dollars (of free credit they had) through "architectural blunders". And the running cost of GPU, CPU, storage, and data transfer summed up to $10K a month - at which point they figured th…

Now, when they make blunders, they don’t see an overt bill for them and are thereby happier.

They also have free heating

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#99

Post by Nextgrid »

pjerem wrote: post #48
Nextgrid wrote: post #38

Earlier quoted context omitted.

Product-specific functionality & configuration. Every cloud provider has their own product, features & configuration for the same basic thing. In the past, if you needed a load-balancer & reverse proxy you'd use Nginx or HAProxy regardless of the underlying machine. Now in the cloud, although you can technically run it on a VM, it's not "best practice" and you should instead reimplement it using your cloud vendor's p…

"In the past" ? My current employer, a high growth company uses HAProxy and Nginx in production with no plan to change anytime soon. Those are not tools from the past. They still work really well and there is no law requiring your brand new startup to be on AWS. Most companies (even more when they are B2B) have very predictable workloads.

> there is no law requiring your brand new startup to be on AWS

See https://news.ycombinator.com/item?id=30272588.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#100

Post by brimble »

treis wrote: post #71
vidarh wrote: post #70

Earlier quoted context omitted.

If you have to use more than one a day a month managing the server, you are doing something wrong. I used to manage multiple racks worth of servers on top of managing the 1k containers running on them, maintaining the (pre-kubernetes) orchestration software I had written to deploy containers to our servers, and still had time left over to spend the majority of my time on the architecture and project management of new…

I'd say if you're not spending one day a month you're doing something wrong. Namely, you're not testing your back ups and disaster recovery often enough.

You're right, but given the context of the thread, it's worth pointing out that's not extra work that one doesn't also need to do with cloud hosting. Ought to test your backups and disaster recovery regularly, either way.
Post reply on HN