Live data from Hacker News

Ask HN: Is your company sticking to on-premise servers? Why?

news.ycombinator.com

91–100 of 782 posts

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#91
We have mostly switched to, or are in the midst of switching to, the cloud.

Services that we will continue to run on-premises (as an exception to that rule) are some machine learning training clusters (where we need a constant, high-level amount of GPU and cloud provider pricing for GPU machines is very far off the mark of what you can build/run yourself) and some file shares for our production facilities where very large raster files are created, manipulated, sent to production equipment, and then shortly afterwards deleted.

Most everything else is going to the cloud (including most of our ML deployed model use cases).

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#92
post #3

Earlier quoted context omitted.

Which, all things considered, seems lower on the cloud. Could you give more details on this answer?

https://techcrunch.com/2017/09/15/why-dropbox-decided-to-dro...

Dropbox has outgrown AWS. I’m not sure it’s a good example.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#93

Mostly, headspace. If I run my own server, I just need to apply my existing Ubuntu sysadmin knowledge. If I use AWS, I have to learn a whole load of AWS-specific domain knowledge, starting with their utterly baffling product names. My time is more valuable than that. Also, sheer cost. Literally everyone I know in my particular part of the industry uses Hetzner boxes. For what I do, it’s orders of magnitude cheaper th…

That’s how you get old, when your time is more valuable than a massive shift in technology.

Nah, we already did mainframes in the 1970s. Renting CPU time only makes sense if you don’t need CPU time or you like wasting money.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#94
post #24
post #13

Earlier quoted context omitted.

How is the cloud less secure than your on-prem servers? I would argue that it's easier to keep track of all the threats with the tools available from big cloud providers.

The big question is - how much do you trust these providers.

I would guess that approximately zero banks own data centers that are operated by their own employees. There might be a few exceptions to this, but the reality is that most banks don't view technology as part of their core business. So this largely gets outsourced to IT consulting firms like Infosys, IBM, Wipro, etc.

The big question is - how much do you trust these providers, and do you think they are more competent at security than Amazon/Google/Microsoft?

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#95

Earlier quoted context omitted.

Can you share the architecture and stack? What form do these jobs have? How do you manage workloads? How do you manage resources? Do users have quota for compute and storage? Do you use GPUs? If so, how do you deal with malfunction? Is this a distributed processing? Are the machines heterogeneous? What do you use for that cluster? What if a job requires dependencies? Do you create a compute environment on the fly or…

I wrote about it here: https://aravindh.net/post/sysadmin/ > What form do these jobs have? Mostly batch jobs written as bash scripts. Occasionally, some users run singularity containers. But, all through SLURM. > How do you manage workloads? SLURM > How do you manage resources? As a sysadmin, my inventory is via Ansible. All activity on servers happen via Ansible only. > Do users have quota for compute and storage? Y…

Fascinating blog post of yours, thanks for sharing it!

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#97
Yes. Three major reasons:

- Cost. It's vastly cheaper to run your own infra (like, 10-100x -- really!). The reason to run in cloud is not to save money, it's to shift from capex to opex and artificially couple client acquisition to expenditure in a way that juices your sheets for VCs.

- Principle. You can't do business in the cloud without paying people who also work to assemble lists of citizens to hand over to fascist governments.

- Control. Cloud providers will happily turn your systems off if asked by the government, a higher-up VP, or a sufficiently large partner.

EDIT: I should add. Cloud is great for something -- moving very fast with minimal staffing. That said, unless you get large enough to renegotiate you will get wedged into a cost deadend where your costs would be vastly reduced by going in-house, but you cannot afford to do so in the short term. Particularly for the HN audience, take care to notice who your accelerator is directing you to use for cloud services -- they are typically co-invested.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#98
post #6

Earlier quoted context omitted.

Interesting. How many machines do you have, roughly? Are you the only sysadmin for this? Also, do you have any public facing services hosted from that infrastructure?

I am the only one administering this cluster. We have approximately 1000 cores across 45 servers. A few public facing services - web servers, small APIs, a web front end tool for a big genetics database. Nothing big. Public services are cordoned off from the compute cluster.

We had a similar setup at our University as well. Execs decided we needed to be a part of school-wide clusters though, and are now trying to retire even that computing hardware and push us to the cloud without covering any of the costs.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#99
post #50

Earlier quoted context omitted.

Hundreds of terabytes a day is really not that much, depends on what latency can you accept. I often run computations over datasets that are petabytes in size, just for my own needs. A big data move would be at least tens of petabytes or more like hundreds, or thousands. Also surprised about latency, latency from what to what? Big cloud providers have excellent globally spanning networks. Long distance networking is…

> Hundreds of terabytes a day is really not that much How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical question. A lot of our servers have 10-40gbit links that we saturate for minutes/hours at a time, which I suspect would be expensive without the kind of topology optimization we do in our datacenters. > Also surprised about latency We've spent a surprising amoun…

> How much would it cost to move this across boxes in EC2?

Nothing. You generally only pay for data going out of cloud providers. Not data going in or data being transferred within the same region.

> One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. This would cost us a lot of money (mostly in dev time, to work around it).

You're not going to have a successful cloud experience unless you build your applications in a cloud suitable way. This means not all legacy applications are a good fit for the public cloud. Most companies really embracing the cloud are mitigating those risks by distributing workloads across multiple instances so you don't care if any one needs to be restarted, especially within a planned window.

> Also, multi-second network dropouts in big cloud datacenters are not uncommon (in my limited experience), but that would be really bad for us. We have millisecond-scale failover with 2x or 3x redundancy on important systems.

Are these inter-region network dropouts or between the internet and the cloud data center? You're not going to be relying on a public internet connection to the cloud for critical workloads.

All that being said, there are plenty of workloads which I don't think fit well in the cloud operating model. You may very well have one of them.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#100
post #50

Earlier quoted context omitted.

Hundreds of terabytes a day is really not that much, depends on what latency can you accept. I often run computations over datasets that are petabytes in size, just for my own needs. A big data move would be at least tens of petabytes or more like hundreds, or thousands. Also surprised about latency, latency from what to what? Big cloud providers have excellent globally spanning networks. Long distance networking is…

> Hundreds of terabytes a day is really not that much How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical question. A lot of our servers have 10-40gbit links that we saturate for minutes/hours at a time, which I suspect would be expensive without the kind of topology optimization we do in our datacenters. > Also surprised about latency We've spent a surprising amoun…

> How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical question.

Data transfer between instances in the same AZ is free. If the data crosses AZs, you're changed $0.01 per GB in both directions. This is for instances on a VPC. I think the pricing model is different for classic EC2.

There are some exceptions like all traffic between EC2 and ALBs being free.

Edit: Pricing is described at https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer

Post reply on HN