Ask HN: Is your company sticking to on-premise servers? Why? • Hacker News Board

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#381

Post by joshuaellinger » Thu, May 07, 2020, 4:45 AM UTC

On the smaller end of the scale, I have a $12K/mo spend with Azure. I decided to go back to Coloc.

For under $50K, I have 4 machines with an aggregate 1TB RAM, 48 cores, 1 pricy GPU, 16TB of fast SSD, 40 TB HHD, and infiniband @ 56GB/sec. Rent on the cabinet is less than $1K/mo. It's going to cost me about $20K in labor to migrate.

So the nominal break-even point is six months but the real kicker is that this is effectively x10-30 the raw power of what I was getting on the cloud. I can offer a quantitatively different set of calculations.

It also simplifies a bunch of stuff: 1. No APIs to read blob data -- just good old files on an ZFS share. 2. No need to optimize memory consumption. 3. No need for docker/k8s/etc to spin up under load. Just have a cluster sitting there.

There are downsides but Coloc beats the cloud for certain problems.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#382

Post by icelancer » Thu, May 07, 2020, 4:45 AM UTC

hprotagonist wrote: post #53

AI compute is so much cheaper on-prem that it's not even in question. And there are clients that demand it. And researchers, in general, like to do totally wacky things, and it's often easier/cheaper to let us if you have physical access.

Yep. This is where we are at.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#383

Post by hn_throwaway_99 » Thu, May 07, 2020, 4:45 AM UTC

RachelF wrote: post #340

mmmBacon wrote: post #276

Earlier quoted context omitted.

A few years ago I was trying to start a company and get it off the ground. We had to make decisions on our tech stack and whether we were going to use AWS and build around their infra. Our business was very data heavy and required transferring large datasets from outside to our databases. Even in our early prototypes, we realized that we couldn’t scale cost-effectively on AWS. I figured out that we could colocate and…

Dropbox did the same thing a few years back - moved everything from Amazon S3 to their own storage. My guess is they did it for cost reasons.

They did, but at Dropbox's size it should be pretty obvious. I mean, their whole business model is essentially acting as a cloud storage provider. Once they got big enough, of course it made sense for them to optimize their infrastructure instead of letting another company take x% off the top.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#384

Post by dwohnitmok » Thu, May 07, 2020, 4:46 AM UTC

eyegor wrote: post #227

j88439h84 wrote: post #197

Earlier quoted context omitted.

I'm curious why you moved to LLNL from HFT?

Money is a safe guess. Research pay scales aren't even close to private sector, especially not finance.

It sounds like the opposite direction happened here.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#385

Post by abathur » Thu, May 07, 2020, 4:46 AM UTC

cwojno wrote: post #369

AgentK20 wrote: post #207

Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…

I worked for a boss that pretty much demanded that we move to the cloud. I showed them the costs for the then 2 providers (GCP/AWS) and arrived at the exact same conclusion on server hosting alone, as bandwidth wasn't the main driver of our application. The rationale was that we'd save so much money by not having to manage the servers ourselves, but we honestly spent much, much more time in software deployments than…

To be clear, it took much more time to deploy your own software to the cloud relative to on-prem?

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#386

Post by tpetry » Thu, May 07, 2020, 4:48 AM UTC

andrenotgiant wrote: post #333

AgentK20 wrote: post #207

Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…

It sounds like you've got a good setup going with colo, but just as a way of illustrating some of the small providers lower bandwidth costs: DigitalOcean gives 3TB bandwidth on a $15 2CPU/2GBMEM/60GBSSD instance. If you ran 1350 of them it'd cost you ~$20k/month and get you your 4PB egress within bandwidth allowance.

Bandwidth Alliance is only for routing http traffic, but a game server will have udp traffic.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#387

Post by brownbat » Thu, May 07, 2020, 4:49 AM UTC

AgentK20 wrote: post #207

Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…

I remember a post here a while back from a guy running Bitcoin mining in his dorm room. One day he realized he could offer his spare cycles to grad students with high computing workloads and undercut cloud computing prices while increasing what he made far over just mining.

It's very weird that we haven't fully arbitraged $/instruction to a single (low) price yet (or storage/hosting, whichever).

If only there were an Uber for unused cycles or storage... let everyone turn their unused capacity into mini AWSs with a common interface and safety and reliability guarantees.

Maybe there are too many barriers, like security.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#388

Post by grantlmiller » Thu, May 07, 2020, 4:53 AM UTC

opportune wrote: post #233

grantlmiller wrote: post #21

I always find it important to separate "cloud" into 2 categories: 1. IaaS - Which I mainly define as the raw programmable resources provided by "hypercloud" providers (AWS, GCP, Azure). Yes, it seems that using an IaaS provider with a VPC can provide many benefits over traditional on-prem data centers (racking & stacking, dual power supply, physical security, elasticity, programmability, locations etc). 2. SaaS - I l…

What about first-party SaaS? Those can also be big features that bring people to some cloud providers. Not all SaaS requires you to trust your data/availability to some random vendor. Of course those first-party SaaS aren't typically suitable for lift-and-shift by their very nature, and they can still have some rough edges, but IMO you can expect them to be almost as reliable as IaaS

First-party SaaS meaning things like RDS, DBaaS, queues, LBs etc? Most of that I would sort of put into a IaaS controlled PaaS, rather than true IaaS SaaS. Yes, these are generally higher on the trust spectrum as they don't involve additional vendors accessing/managing/storing data.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#389

Post by Mave83 » Thu, May 07, 2020, 4:57 AM UTC

Cloud is only good if you don't care about costs and plan to scale without looking back.

For example, building a Ceph based Software Defined Storage with croit.io for S3 comes for 1/10 to 1/5 of the AWS price in TCO. Same goes for any other product in the cloud.

If you only need it a short time up to 6 months, go to the cloud. If you plan to have the resources longer than 6 months go to Colocation.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#390

Post by dasil003 » Thu, May 07, 2020, 4:59 AM UTC

horsawlarway wrote: post #217

I'm slowly coming to the complete opposite opinion you seem to have. I've worked almost entirely for companies that run services in various cloud infrastructures - Azure/Heroku/Aws/GCP/Other. I recently started a tiny 1 man dev shop in my spare time. Given my experience with cloud services it seemed like a no brainer to throw something up in the cloud and run with it. Except after a few months I realized I'm in an in…

IMHO the optionality the cloud gives you is the main value at the low-end. You can play around with different architectures, mixing and matching managed services with no commitment. When you need to scale you can always switch to on-prem, and with the benefit of having a working system to base your hardware requirements on. Of course this migration has costs too, but if this is your strategy you can minimize cloud-specific dependencies and leverage automation that can be used in both environments (eg. ansible).