Ask HN: Is your company sticking to on-premise servers? Why? • Hacker News Board

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#311

Post by mayank » Thu, May 07, 2020, 2:55 AM UTC

mmmBacon wrote: post #292

mayank wrote: post #282

Earlier quoted context omitted.

You’re not paying Google $5B for raw infra, you’re paying for cloud services like top-tier horizontally scalable databases, global availability, CDNs, datastores of different flavors, and transparently managed monitoring and hardware fault resolution.

Yes, you’re paying for a ton of stuff there that you probably don’t use and then are susceptible to bugs that have nothing to do with your use case. At $5B it would not cost anywhere near that much to replicate. Your infra would be better tailored to your workloads and your sw teams which would further drive costs down. The upsides are that you only drive features you need and keep things simple. The downside is that…

> Yes, you’re paying for a ton of stuff there that you probably don’t use

This is the antithesis of any cloud. You only pay for what you use.

> At $5B it would not cost anywhere near that much to replicate.

If you can recreate GCP for $5B in capex, there are likely some VCs lurking here who would like a word with you.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#312

Post by boulos » Thu, May 07, 2020, 2:59 AM UTC

mmmBacon wrote: post #276

AgentK20 wrote: post #207

Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…

A few years ago I was trying to start a company and get it off the ground. We had to make decisions on our tech stack and whether we were going to use AWS and build around their infra. Our business was very data heavy and required transferring large datasets from outside to our databases. Even in our early prototypes, we realized that we couldn’t scale cost-effectively on AWS. I figured out that we could colocate and…

Disclosure: I work on Google Cloud.

> paying Google $5B

You were off by 10x :). The annual commitment was $400M/yr on average. Snap’s S1 [1] said:

> We have committed to spend $2 billion with Google Cloud over the next five years

[1] https://www.sec.gov/Archives/edgar/data/1564408/000119312517...

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#313

Post by ablekh » Thu, May 07, 2020, 3:00 AM UTC

sseagull wrote: post #283

ablekh wrote: post #264

Earlier quoted context omitted.

I appreciate your comment and links (BTW, I'm slightly familiar with NWChem and similar software :-). Having said that, I'm not sure how your comment addresses my question. You are talking about internally-developed software transfer from the lab to external users, whereas my use case is delivering externally-developed commercial software to national labs' users, a value transfer in the opposite direction. Care to co…

I can speak at least a little bit about my field (computational chemistry). A lot of academic software is free and/or open source (like NWChem). These are installed by the cluster support for use by anyone. This is helpful because the software can be optimized beyond what most end users can do. Beyond that, a lot of academics run custom software or uncommon packages. By definition, many academics are doing something…

I appreciate your feedback. My question above is not about the mechanics of deploying commercial scientific software (it certainly somewhat varies between fields and organizations, though common themes and methods are pretty standard). It is about the procurement processes within national labs ecosystem. That is, about governmental gating / filters "in front of" vendors' go-to-market strategy and processes (RFPs, pilots etc.).

Re: more details - I'm not working in computational chemistry per se, but in the adjacent and closely related, as you understand, field of materials science and engineering. Will be happy to discuss my plans (within limits - currently very early and in stealth) through direct channels. You can connect with me on LinkedIn or shoot me an e-mail (see above).

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#314

Post by evanb » Thu, May 07, 2020, 3:00 AM UTC

tgamblin wrote: post #104

I work in Livermore Computing at LLNL. We manage upwards of 30 different compute clusters (many listed here: https://hpc.llnl.gov/hardware/platforms ). You can read about the machine slated to hit the floor in 2022/2023 here: https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capita... . All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. F…

I'm a former LLNL lattice QCD postdoc and current user of the LLNL machines (and others). I have to say, I was wildly spoiled.

Every other computing center where I have an account cannot hold a candle to the helpfulness of the LLNL support, the ease of use (except for the 2-factor requirement, which I understand), and absolutely insane power.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#315

Post by busterarm » Thu, May 07, 2020, 3:01 AM UTC

I'm going to buck the trend and say cloud is great. We do cloud, on-prem and colo (16 racks in two different DCs).

Procurement is a nightmare especially when your vendor is having problems with yields (thanks Intel!) and the ability to scale up and scale down without going through hardware procurement process saves us millions of dollars a year.

We avoid the lock-in by running on basic services on multiple cloud providers and building on top of those agnostically.

Spend is in the millions per month between the cloud providers, but the discounts are steep. We're essentially had to build our own global CDN and the costs are better than paying the CDN services and better than running our own hardware & staffing all those locales.

It's a no brainer. We'll continue to operate mixed infrastructure for quite some time as certain things make sense in certain places.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#316

Post by mathattack » Thu, May 07, 2020, 3:01 AM UTC

We are going to the cloud, but you have to be careful. With on-prem the limit of the cost is the server. Someone writes inefficient code and it just doesn’t work. In the cloud there are 1000 ways to overspend and the vendors purposefully don’t make it easy to track or keep things under control.

It’s kind of like outsourcing. If you don’t know what you are doing, cost goes up and quality goes down.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#317

Post by _bxg1 » Thu, May 07, 2020, 3:05 AM UTC

One of my favorite things (at least for personal projects) about using the cloud is so-called "platform as a service" systems like Heroku, where I don't have to get down in the weeds, I just push code and the process starts (or restarts).

Is there something like that I could use on my own hardware? I just want to do a fresh Linux install, install this one package, and start pushing code from elsewhere, no other configuration or setup necessary. If it can accept multiple repos, one server process each, all the better. I know things like Docker and Kubernetes exist but what I want is absolute minimal setup and maintenance.

Does such a thing exist?

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#318

Post by mbreese » Thu, May 07, 2020, 3:05 AM UTC

reacharavindh wrote: post #25

Jugurtha wrote: post #17

Earlier quoted context omitted.

Can you share the architecture and stack? What form do these jobs have? How do you manage workloads? How do you manage resources? Do users have quota for compute and storage? Do you use GPUs? If so, how do you deal with malfunction? Is this a distributed processing? Are the machines heterogeneous? What do you use for that cluster? What if a job requires dependencies? Do you create a compute environment on the fly or…

I wrote about it here: https://aravindh.net/post/sysadmin/ > What form do these jobs have? Mostly batch jobs written as bash scripts. Occasionally, some users run singularity containers. But, all through SLURM. > How do you manage workloads? SLURM > How do you manage resources? As a sysadmin, my inventory is via Ansible. All activity on servers happen via Ansible only. > Do users have quota for compute and storage? Y…

Yeah, cloud dev-ops and HPC are so similar and still oh-so different (I run a smaller HPC cluster for genomics). When reading these questions, someone with an HPC background would probably already know the answers (or just accept SLURM as an answer for everything).

With respect to the original questions -- I've investigated running a full SLURM cluster on the cloud and it just never seems worth the effort. For larger clusters than mine, maybe, but then when you start to hit the levels you're talking about, I just don't see the point in moving to the cloud. It would be hard to hit that sweet spot where the costs would make sense. Amazon even has a series on scaling a SLURM cluster with AWS EC2 provisioning, but it just seemed like more work than was justified for us.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#319

Post by kelnos » Thu, May 07, 2020, 3:06 AM UTC

YarickR2 wrote: post #269

RhodesianHunter wrote: post #237

Earlier quoted context omitted.

> Cloud is great if your workload is variable and erratic Or if you're just constantly iterating on a large product with many engineers. Those engineers' salaries almost always outweigh all of your cloud costs and so making them productive is cost effective. Things like SNS/SQS/S3/VPC/ELB/etc. save you countless hours and often make up for the increased cloud costs with increased developer productivity.

No they don't . Things you mentioned are very easy to deploy on-premises, and they won't take more than one man month to deploy and maintain.

That's an incredibly naive assessment and is absolutely not true in practice, except for perhaps for some orgs where the stars just happen to align to make that a reality. Most places heavily build "cloud" into their frameworks, tooling, hiring, and, well, everything. Even just swapping out a single hosted/managed component (as my company did a few years ago, replacing Kinesis with Kafka) can require a lot of up-front work[0] and on-going maintenance.

[0] Especially if you're already in production with the thing you want to replace, and want to transition without downtime for your customers.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#320

Post by nikisweeting » Thu, May 07, 2020, 3:07 AM UTC

Quebec power and internet pricing is really competitive. For residential services I pay $0.06/kw + $80/mo for 1Gbps fiber with 1ms ping to 8.8.8.8 (USD).

As a result, I run a power-hungry Dell r610 with 24 cores and 48GB of ram with 20+ services on it for many different aspects of my company. All the critical stuff runs on DigitalOcean / Vultr, but the 20+ non-critical services like demo apps, CI/CD, cron workers, archiving, etc. run for <$200/yr in my closet.