Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is your company sticking to on-premise servers? Why?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 782 posts

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#101

Post by bcrosby95 »

We have around 20 servers in a colo center down the street.

At this number of servers we can still host websites that have millions of users (but not tens of millions). They are not exotic servers either. In fact by now they are, on average, around 11 years old. And costed anywhere from 2k to 8k at the time of purchase. Some are as old as 19 years. Hell, when we bought some of them - with 32GB of memory each - AWS had no concept of "high memory" instances and you had to completely pay out your ass for a 32GB server, despite ram being fairly cheap at the time.

We have no dedicated hardware person. Between myself and the CTO, we average maybe a day per month thinking about or managing the hardware. If we need something special setup that we have no experience in, we have a person we know that we contract, and he walks us through how and why he set it up as he did. We've used him twice in the last 13 years.

The last time one of us had to visit the colocation center was months ago. The last time one of us had to go there in an emergency was years ago. It's a 5 minute drive from each of our homes.

So, why exactly should we use the cloud? We have servers we already paid for. We rent 3 cabinets - I don't recall the exact cost, but I think its around $1k per month. We spend practically no time managing them. In our time being hosted in a colo center - the past 19 years - we've had a total of 3 outages that were the fault of our colo center. They all lasted on the order of minutes.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#102

Post by frakkingcylons »

sneak wrote: post #56
centimeter wrote: post #36

We are a 1000-2000 person company and we have probably on the order of $100M of servers and data centers and whatnot, and I think we spend about 2/3rds of that every year on power/maintenance/rent/upgrades/etc. We don't generally trust cloud providers to meet our requirements for: * uptime (network and machine - both because we are good at reliability [and we're willing to spend extra on it] and because we have lots…

> We also have a number of machines with specs that would be hard to get e.g. on AWS. What specs are those? I was under the impression AWS has everything from extremely tiny to giant terabytes-of-ram-for-SAP instance types.

It could be that they need hardware that can support less common architectures, like Solaris or AIX.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#103

Post by toast0 »

alephnan wrote: post #81
toast0 wrote: post #46

Earlier quoted context omitted.

The cloud provider has physical access to the machines. They may have software access to the machines. I'm sure they're contracted not to do anything bad, but there is risk of an insider incident at the cloud provider leading to an incident the customer can't prevent.

Encryption?

Doesn't help if the provider snapshots the ram containing the keys.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#104

Post by tgamblin »

I work in Livermore Computing at LLNL.

We manage upwards of 30 different compute clusters (many listed here: https://hpc.llnl.gov/hardware/platforms). You can read about the machine slated to hit the floor in 2022/2023 here: https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capita....

All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. For our workloads on "commodity" x86_64/no-GPU clusters, we pay 1/3 or less the cost of what you'd pay for equivalent cloud nodes, and for the really high end systems like Sierra, with NVIDIA GPUs and Power9's, we pay far less than that over the life of the machine.

The way machines are procured here is different from what smaller shops might be used to. For example, the El Capitan machine mentioned above was procured via the CORAL-2 collaboration with 2 other national labs (ANL and ORNL). We write a 100+ page statement of work describing what the machine must do, and we release a set of benchmarks characterizing our workload. Vendors submit proposals for how they could meet our requirements, along with performance numbers and test results for the benchmarks. Then we pick the best proposal. We do something similar with LANL and SNL for the so-called commodity clusters (see https://hpc.llnl.gov/cts-2-rfi for the latest one). As part of these processes, we learn a lot about what vendors are planning to offer 5 years out, so we're not picking off the shelf stuff -- we're getting large volumes of the latest hardware.

In addition to the cost savings from running on-prem, it's our job to stay on the bleeding edge, and I'm not sure how we would do that without working with vendors through these procurements and running our own systems.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#105

Post by arghwhat »

Quite frankly, "cloud" is a convenience and elasticity service at a steep premium, with downsides.

Contrary to popular belief, it does not in the slightest save you a sysadmin (most just end up unknowingly giving the task to their developers). And contrary to popular belief, the perf/price ratio is atrocious compared to just buying servers.

For some of the loads I had been doing the math for, I could rent a colo and buy a new beefy server every year with money to spare for the yearly cost of something approximating the performance in AWS...

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#106

Post by Polylactic_acid »

snarfy wrote: post #83

The idea of the cloud is to only pay for what you use. Your on-premise server is idle 99% of the time so why are you paying for a full server? If that's not true, it turns out it's quite expensive to run things in the cloud. If your workload is crunching numbers 24/7 at 100% cpu, it's better to buy the cpu than to rent it.

Cloud servers tend to be more reliable as well if you don't run your own datacenters. We have lost our internet connection or power 3 times in the last year in the office. Its not the end of the world since we can go to 4g for our own usage but if our servers were hosted locally this would be a huge issue.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#107

Post by hkmurakami »

centimeter wrote: post #36

We are a 1000-2000 person company and we have probably on the order of $100M of servers and data centers and whatnot, and I think we spend about 2/3rds of that every year on power/maintenance/rent/upgrades/etc. We don't generally trust cloud providers to meet our requirements for: * uptime (network and machine - both because we are good at reliability [and we're willing to spend extra on it] and because we have lots…

Sort of sounds like why some large users will opt to go with a wholesaler like Digital realty, or maybe even Equinix.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#108

Post by jasonv »

janstice wrote: post #87

I'm application architect at enterprise-type org. We have a few SaaS applications, but all the big dogs, including custom dev, run in-house on a dual data centre vmware environment. It's cheaper for us to spin up more VMs in-house, so there's no real cloud driver for things that just live on VMs. On the other hand, our ops team are still working on network segmentation and self-service, but I regularly get a standard…

Last company I was at was more or less as you describe.

Now.. at a company in a different industry, there's a 5+ year plan to move 100% to cloud. Nascent efforts are about 18 mos old already, no apps are live yet.

Fortunately, they've been using a container approach for their on-prem stuff for a while, so some stuff can move over pretty easily, a lot of things will get a touch-up or more interesting upgrade along the path to the cloud environment.

Not even talking about de-commissioning the DCs yet, but those will get defunded as things go on.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#109

Post by H8crilA »

centimeter wrote: post #52
H8crilA wrote: post #50

Earlier quoted context omitted.

Hundreds of terabytes a day is really not that much, depends on what latency can you accept. I often run computations over datasets that are petabytes in size, just for my own needs. A big data move would be at least tens of petabytes or more like hundreds, or thousands. Also surprised about latency, latency from what to what? Big cloud providers have excellent globally spanning networks. Long distance networking is…

> Hundreds of terabytes a day is really not that much How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical question. A lot of our servers have 10-40gbit links that we saturate for minutes/hours at a time, which I suspect would be expensive without the kind of topology optimization we do in our datacenters. > Also surprised about latency We've spent a surprising amoun…

> A lot of our servers have 10-40gbit links that we saturate for minutes/hours at a time, which I suspect would be expensive without the kind of topology optimization we do in our datacenters.

I think everyone does something of this sort nowadays, that's why networking is ~free within data centers :)

> but an extra 1ms (say) between datacenters is generally bad for us and measurably reduces performance of some systems

That speaks to me. You will always be just the n-th client unless you own the cross-datacenter data links (i.e. have full autonomy on deciding the priority of the traffic). It's similar to the covid provisioning problems you had mentioned.

> One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box.

Yeah like others pointed out - that's just what "cloud" is, and is generally a good idea. You're supposed to handle a certain % of your machines going dark without a warning without violating any SLO (or even worse, certain % of your machines "pretending" they're up but actually being ridiculously slow for this or that reason; and don't even get me started on CPU/RAM bitflips).

It sounds to me that you run an extremely highly sensitive service, something for which paying for true ownership of the hardware just makes sense to remove those kinds of risks that most services don't care about. At the end of the day "cloud" is a shared resource, and no resource separation efforts will be 100% effective.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#110

Post by Guest42 »

jm4 wrote: post #73
centimeter wrote: post #36

We are a 1000-2000 person company and we have probably on the order of $100M of servers and data centers and whatnot, and I think we spend about 2/3rds of that every year on power/maintenance/rent/upgrades/etc. We don't generally trust cloud providers to meet our requirements for: * uptime (network and machine - both because we are good at reliability [and we're willing to spend extra on it] and because we have lots…

What kind of business is that? That level of spending seems insane based on headcount. At least without knowing anything about the business. Are you running public facing services or something? What does that look like as a percentage of revenue?

Working at a medium-sized bank the data center costs were very significant.
Post reply on HN