Live data from Hacker News

Ask HN: Is your company sticking to on-premise servers? Why?

news.ycombinator.com

421–430 of 782 posts

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#421

Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…

> Cloud is great if your workload is variable and erratic

I would say also, the cloud is cheap if you can shut down or scale-down services when you need it.

IMHO if you ow your software stack, you can use the cloud to rationalize your spending (for instance, Netflix changed their default codec some weeks ago which made them save a lot of monet in egress bandwith)... but nobody can do this.

If your company runs prepackaged software, like an SAP, Manhattan, you don't have margin to shutdown services when they're not needed (at least by now).

Also, after a long time working in a datacenter, I am still ashamed how powerful bare-metal has become: For less than 30K you can get a 1TB server (without storage) with at least 40 cores. Some vendors are even offering hardware as Opex in a pay-as-you-go seting to compete against the cloud.

(disclaimer I work for a Google Cloud partner)

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#422
In physics, a gallon of water weights 8.34 lbs. (For this analogy, a gallon of water is a unit of work.) And the gallon of water weighs 8.34 lbs irregardless if it is sitting on my desk in a physical building, or on your desk, in the cloud. Same weight, same unit of work. same effort. For a brand new, greenfield application, the cloud is a no brainer. I agree 100%. But for legacy applications, and there are so, soooo many, the cloud is just some one else's computer. Yes, the cloud is more scaleable, yes, the cloud is more manageable, and yes, you can control the cpu/storage/memory/network in much finer amounts. But legacy applications are very complicated. They have long tails, interconnections to other applications that cannot immediately be migrated to the cloud. I have migrated clients off of the cloud, back to on premise or to (co-lo) local hosting, because without rewriting the legacy application, the cloud costs are simply too great.

The essence of IT is to apply technology to solve a business problem. Otherwise, why would the business spend the money? The IT solution might be crazy/stupid/complex but if it works, many business simply adopt it and move on. Now, move that crazy/stupid/complex process to the cloud and surprise, it is very, very expensive. So, yes, the cloud is better, but only for some things. And until legacy applications are rewritten on-premise will exist.

One final insight. The cloud costs more. It has been engineered to be so, both from a profitability standpoint(Amazon is a for profit company) but also because the cloud has decomposed the infrastructure of IT into functional subcomponents, each of which cost money. When I was younger, the challenge for IT was explaining to management, the ROI of new servers, expanded networking, additional technology. We never quite got it right and often had it completely wrong. That was because we lacked the ability to account for/track and manage the actual costs of an on-premise operation. Accounting had one view, operations had another view and management had no idea really, why they were spending millions a year and could not get their business goals accomplished. The cloud changed all of that. You can do almost anything in the cloud, for a price. And I will humbly submit, that the cost of the cloud - minus the aforementioned profitability, is what on-premise organizations should have been spending all along. Anyone reading this and who has spent time in a legacy environment, knows that it is basically a futile exercise of keeping the plates spinning. On-premise failed because it could not get management to understand the value on in-house IT.

As I said, the costs are the same. A gallon of water weighs what it weighs regardless of location. It will be interesting to see, I predict the pendulum will swing back.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#423
My last three employments have had me and my team build three different platforms with three different “providers”. Many lessons learned!

Chronological order:

1. E-commerce, low volume (1000-5000RPM), very high value conversions, highly localized trade.

We built an on-prem stack using hasicorp here. This place had on-prem stuff already in place, the usual vendor driven crap - expensive hypervisor, expensive spoffy SAN, unreliable network. Anyway, my platform team (4-5 guys) built a silo on commodity hardware to run the new version of this site. This is a few years back, but the power you get from cheap hardware these days is astounding. With 6 basic servers, in two DCs, stuffed with off the shelf SSDs we could run the site and dev teams no problem. Much less downtime compared to the expensive hyperconverged blade crap we started on at basically no cost. There’s a simplicity that wins out using actual network cables and 1u boxes... LXC is awesome btw! Using “legacy” vmware, emc, hp etc for non-essential on-prem? Cloud is tempting!

2. Very high volume (Billions of requests per day), global network. AWS. Team tasked with improving on-demand scalability. We implemented kubernetes on AWS and it really showed what it’s about! After 6-7 months of struggle with k8s 3. Upstart dev-shop. No rpm outside dev (yet). Azure. About 30 devs building cool stuff. Azure sucks as IaaS, they want you to PaaS that’s for sure. Cloud decision had been made already when I joined. Do you need cloud for this? No. Are there benefits? Some. Do they outweigh the cost? Hardly. In the end it will depend on how and where your product drives revenue. We pay for a small local dev datacenter quarterly, which i find annoying.

Just some quick thoughts off the top of my head (on the phone so excuse everything).

Happy to discuss further!

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#424

Earlier quoted context omitted.

Same here - CTO of medium sized company. Our IT infra costs are 1/10th the cost of cloud, simply because I happen to be comfortable having on-premise machine and working on them (sometime myself). We have two dozen servers in two locations. It's more time to setup, but maintenance is actually quite low.

Is this the all in cost? Including lease on land/building, staff, electricity etc...

[deleted]

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#425

Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…

Cost. We currently pay less than €5000 monthly for 500TB/month in traffic and 50 Ryzen CPUs. Amazon would be $30,000 traffic + $100,000 compute.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#426
For business logic at a fairly large school the free and open tools that make the cloud so productive get used here a lot. We get to leverage commodity hardware and network infra very effectively for on-premises[1].

You have to have a good recovery plan for when equipment X’s power supply fails but when deploy is all automated, it’s very easy to overcome swapping bare metal, and easy to drill (practice) during off hours.

This makes it much easier to meet regulatory compliance: either statutory or regulations your org has created internally (e.g. financial controls in-org, working with vulnerable people or children, working with controlled substances, working with sensitive intellectual property.)

Simply being able to say you can pull the plug on something and do forensic analysis of the storage on a device is an important thing to say to stakeholders (carers, carers families, pupil parents.)

I’m so grateful to be living in the modern age when “cloud” software exists[2], but I don’t have to be in the cloud to use it.

The downside: you need trained staff and it’s completely inappropriate if you need any kind of bandwidth, power consumption, or to support round the clock business (which we do not because, out here on the long tail, we work in single city so still have things like evenings weekends and holidays for maintenance!)

— [1] Premise vs premises is one of those “oh isn’t the English language awful” distinctions. Premise is the logical foundation for some other theory (“the premises for my laziness is that because the sky is grey it will probably rain so I’m not going to paint the house”) where as the premise_s_ means physical real estate property (“this is my freshly painted house: I welcome you onto the premises”.)

[2] Ansible, Ubiquiti, arm SBCs like raspberry pi, Docker, LXC, IPv6 (making global routing for more tractable, IPv4 for the public and as an endpoint to get on the VPN.)

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#427

Earlier quoted context omitted.

You pay for cross-AZ Traffic in AWS, and that adds up really fast.

Yes. You've got to be aware of where those boundaries are when adopting the cloud and the cost information around these cases are inadequate at best. Too many people get surprise bills.

The first step is to stop blaming the victim.

It's nobody's fault that the billing structure at Amazon is so complicated and confusing.

Except Amazon's.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#428

Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…

what i have generally seen is that storage - blob (s3) or database - is the blocker. Even if you go to k8s, the hassle to manage storage is non-trivial.

And because of this singular fact, startups cannot move to on-premise. You really dont want to manage snapshots, restores, backups.

Anyone can manage application servers.

Post reply on HN