Live data from Hacker News

We decided to move 90% of our workload from the cloud to on-prem infrastructure

medium.com

71–80 of 218 posts

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#71
post #70
post #43

Earlier quoted context omitted.

> The most expensive part is the electricity. No, the most expensive part is the persons time for managing it. I can rent a monstrous Dedicated Server for $400/month from OVH, but even with a UK salary, if I have to spend more than 1 day a month on it in any shape or form (and that includes the initial setup), it's cheaper to use "the cloud" or some form of a managed service.

If you have to use more than one a day a month managing the server, you are doing something wrong. I used to manage multiple racks worth of servers on top of managing the 1k containers running on them, maintaining the (pre-kubernetes) orchestration software I had written to deploy containers to our servers, and still had time left over to spend the majority of my time on the architecture and project management of new…

I'd say if you're not spending one day a month you're doing something wrong. Namely, you're not testing your back ups and disaster recovery often enough.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#72
post #4

I hate the fact that hiring now basically requires cloud experience specific to vendors. This is basically going to force people into one of the three major cloud platforms.

There's enough jobs where they're okay with any cloud experience. You might lose access to thousands of jobs by not having Azure on your resume, but there's tens of thousands where it isn't a strict requirement.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#73

I work in HPC for a cloud provider, and fully endorse this move. Anonymously, of course. You can make an economic argument for or against cloud in practically every IT domain, but in HPC the case for on-prem is really compelling; none of the cloud networking/resiliency value-add is relevant to batch workflows, and costs per core-hour are only remotely comparable if you use spot - which is itself a major compromise. T…

As an addendum to this: if you absolutely must use cloud, stick with AWS. Using Azure is (IMO) a fucking miserable experience and their only advantage (InfiniBand) is better served by buying your own hardware. GCP and OCI might be fine if you are getting a lot of credits, but the skills will not be useful down the line - while AWS is expensive, you will at least learn a bunch of in-demand operational skills.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#74
post #43

Earlier quoted context omitted.

Yes, they are cheap. Running one's own server is also easy peasy; far too many think it is difficult, it is not. The most expensive part is the electricity.

> The most expensive part is the electricity. No, the most expensive part is the persons time for managing it. I can rent a monstrous Dedicated Server for $400/month from OVH, but even with a UK salary, if I have to spend more than 1 day a month on it in any shape or form (and that includes the initial setup), it's cheaper to use "the cloud" or some form of a managed service.

I hear that said a lot. I don’t think it stands.

A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t.

The cloud actually requires more investment for large organisations. Previously you might of only had a handful of sysadmins but now you have a large dedicated platform team doing devops type work building your own abstractions/paas on top of your cloud.

The advantage the cloud has is flexibility. You don’t need to go through a lengthy process to acquire hardware. Likewise cloud services are disposable, no longer need something? Hit the delete button.

I think the more PaaS like services such as Heroku, Lambda, Fargate, Google Cloud Run do better realise the less maintenance story but not cloud generally.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#75
post #14

Earlier quoted context omitted.

I'd say don't even consider colo unless you have a specific use case. Rented dedicated servers are cheap, let someone else take care of the hardware.

Yes, they are cheap. Running one's own server is also easy peasy; far too many think it is difficult, it is not. The most expensive part is the electricity.

It’s pretty hard to generalize this without qualifiers. Electricity can be the most expensive problem, but it requires carefully planned control of the other factors - e.g large scale with highly automated servers, network, and meat-reducing control planes to become true. Otherwise factors such as people and under-utilization can especially dominate smaller and/or less efficient facilities.

Thinking thru the factors, many seem obvious, but are often forgotten/ignored when comparing rental or IaaS costs.:

Space is a fixed cost driven by market rates and maximum Server Capital Costs i.e. floor space. Failing to fill the room increases your cost/server efficiency. Pretty common to run out of thermal/power before you run out space, as equipment efficiency increases through the lifespan of the facility.

Server and Power costs scale together, carry a minimum cost for keeping machines on, and vary based on utilization. Again if the servers aren’t doing work, your efficiency ratio will drop. Larger space typically have pre negotiated power commitments too - failing to consume that carries fiscal penalties. Servers unit costs are fairly cheap, storage not so much. Full utilization throughout capital/lease lifespan is the goal - anything less increases relative cost/server.

Network costs scale with Server Costs, and vary again by utilization - minimum invest rules apply, all servers need at least one network port, as well as upstream Core/TOR/Miniswitch gear. The network gear lifespan is typically longer than servers, but shorter than facilities. It usually incurs annual support/maintenance charges too. Bandwidth charges are variable as expected.

People costs scale with a step-function and numbers driven by minimum coverage requirements, task complexities, and level of human toil. Performing any task on a device by-hand is expensive in most markets - touches on tickets, change management, task time etc. Fully burdened S+R in Western cultures is typically ~2x the salary - a $80K employee probably costs around $150K by the time all the workplace costs, taxes, and benefits are paid. Network folk are typically premium resources compared to DC Ops. Sustainable 24x7 coverage looks like a staff of 3-4 people.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#76
post #3

After spending $100k in a year.. After doing the math, We decided to purchase three workstations at a total cost of $17k. One is a GPU-based workstation with two RTX 3090s and an Intel i9–12900 CPU, and another two workstations with 16 cores AMD Ryzen 5950X CPUs. It took us a few FTE days to set those up to our satisfaction with slurm, NFS, backups, and several other services. We noticed that our RTXs, although consi…

We noticed that our RTXs, although considered gaming cards, are comparable (if not better) in performance to Tesla V100

You don't buy Tesla cards because you care about maximizing operations pr second, you buy them because you care about operations pr KWh.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#77

Earlier quoted context omitted.

> They did the math. And bought 3 workstations instead of 3 servers. IMHO the headline of "Why Enzymit Decided to Build its Own On-Prem HPC Infrastructure" is a bit... stretched.

This is how universities (or people from academic background) interpret...

They have a workload that is entirely dictated by their protein design etc. projects, not by external access to a web app. So "high performance" means doing a single definite task quickly (so they can *proceed to the next run), not scaling to many users and requests. If they need to scale, they first hire protein designers or the like and then they set up some new workstation, probably something different from existing ones for diversification and obsolescence reasons. No cattle.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#78

I work in HPC for a cloud provider, and fully endorse this move. Anonymously, of course. You can make an economic argument for or against cloud in practically every IT domain, but in HPC the case for on-prem is really compelling; none of the cloud networking/resiliency value-add is relevant to batch workflows, and costs per core-hour are only remotely comparable if you use spot - which is itself a major compromise. T…

If you can setup HPC at scale you can setup Ceph for object storage. It will save you so much money in the long run.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#79

I work in HPC for a cloud provider, and fully endorse this move. Anonymously, of course. You can make an economic argument for or against cloud in practically every IT domain, but in HPC the case for on-prem is really compelling; none of the cloud networking/resiliency value-add is relevant to batch workflows, and costs per core-hour are only remotely comparable if you use spot - which is itself a major compromise. T…

I'm curious what people are spending / over spending all their money on in the cloud?

My exposure to the actual granular costs and billing have only been limited to a small company and in that case the costs were pretty appealing compared to running everything yourself. Granted this was also a bit of a hybrid with some services local and others in the cloud.

I've not had much exposure to where the deep costs start to pile up as far as cloud services goes. I wonder where those pop up?

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#80

> "Starting a web-based or SaaS (Software as a Service) business was virtually unheard of before the age of IaaS" Nonsense. There were plenty of SaaS startups. There was even a little event called the dotcom boom all about internet companies. This lack of history and experience is why new companies get into this cloud-first mess in the first place. Cloud is primarily for flexibility in iteration, dynamic scaling, or…

In addition: long before AWS you could easily rent virtualized or dedicated servers. It took longer to provision than EC2 and all you had for storage is a fixed amount of disk, but that was absolutely sufficient for many businesses.

Hell, it still is sufficient for many businesses.
Post reply on HN