Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We decided to move 90% of our workload from the cloud to on-prem infrastructure

medium.com

Post reply on HN
Print view View original

141–150 of 218 posts

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#141

Post by bob1029 »

rhplus wrote: post #129
bob1029 wrote: post #2

> We do not have (yet) any public-facing applications that need to scale across multiple geographical zones and handle millions of requests per minute. Most don't. 1mm requests per minute is very pedestrian for a single vm in virtually all cases. 1mm per second is totally reasonable too if you are careful with a few things... I genuinely believe you could put the literal public Netflix biz experience on a single VM.…

1 million/sec is basically line speed on a 10Gbps link if each request is coming in at MTU of 1500 bytes. Sure, you might be able to push that much data through a VM on a test bench with well-behaved local clients, but you ain’t gonna be doing that rate once you add TLS, authz, logging, throttling, non-trivial serialization, non-trivial database access, A/B tests, metrics, fraud detection, recommendations, and everyt…

1500 bytes is a pretty big average payload size when you consider information theory and what actually must be communicated for this kind of business (on average).

A user clicking "Watch later" on a video could theoretically be communicated in something as small as 64-bit integer for the user/session id, one for the command type, and another for the identity of the actual video. With serialization, padding, etc., you are still probably well under 50 bytes for this one event.

Being sloppy with data throughout is certainly a good reason to need more pipes and servers. With enough discipline, you can process events at rates far exceeding 1 million per second with a single box and non-exotic network stack.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#142

Post by secabeen »

SpaceMartini wrote: post #128
Tsarbomb wrote: post #78

Earlier quoted context omitted.

If you can setup HPC at scale you can setup Ceph for object storage. It will save you so much money in the long run.

I have heard too many horror stories about Ceph (and OpenStack) to be confident about that. I certainly don't think I can truly beat S3 on cost or performance at the terabyte scale for household data - and while larger scale would give on-prem savings there are also higher expectations (in terms of availability and performance) of a multi-perabyte storage array.

Really depends on your scale. At the Terabyte to 100s of TB level, you can solve most storage problems at minimum cost with NAS or ZFS on commodity hardware.

Ceph/Object storage comes into its own at the multi-petabyte and higher levels, which is not very many groups or institutions.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#143

Post by lnwlebjel »

SpaceMartini wrote: post #69

I work in HPC for a cloud provider, and fully endorse this move. Anonymously, of course. You can make an economic argument for or against cloud in practically every IT domain, but in HPC the case for on-prem is really compelling; none of the cloud networking/resiliency value-add is relevant to batch workflows, and costs per core-hour are only remotely comparable if you use spot - which is itself a major compromise. T…

> The only real advantage cloud has for science is object storage, which is genuinely a much better idea than trying to manage your own long-term archival storage.

I work with an academic HPC group, and because researchers generally pay only for the hardware, and maybe some recharge rate for occasional maintenance, the cost per TB per month for 100's of TB and larger systems works out to the same for Glacier Deep (about $1/TB/mo) - except there is no 180 day requirement, no egress fees, and no transfer fees. And disk just keeps getting cheaper.

I'm told that big part of the solution is their use of ZFS.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#144

Post by ludjer »

phonon wrote: post #114
post-it wrote: post #104

Earlier quoted context omitted.

The biggest thing for me is no surprise bills. Sure, it's pretty unlikely that a t2.small instance and its associated storage and network subscriptions are going to produce a $1000 bill one day, but there's literally no way to set a hard cap on billing so clearly Amazon thinks it might be possible.

Use https://aws.amazon.com/lightsail/pricing/ then...

At that point you might as well go with vps. Lightsale is way more expensive and the underlying architecture is way more unstable I have a 1% failure rate for my fleet of instances. Ever two weeks I have a instance is not responding and needs to be rebooted alert from aws.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#145

Post by yjftsjthsd-h »

GTP wrote: post #120
philliphaydon wrote: post #88

Earlier quoted context omitted.

> A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t. Ani. If the hardware you’re running on is dying. In ec2 you stop it and start it. It’s on new hardware. If you run bare metal you’re screwed. Disk is dying ? Don’t matter cos your data exists multiple times over in AWS elastic storage. With bare metal y…

>Disk is dying ? Don’t matter cos your data exists multiple times over in AWS elastic storage. With bare metal you got to shut down and replace. Not really, this is what RAID is for

Depending somewhat on the quality of the hardware in question. Hot swap capable isn't rare, but it's not going to happen at the bottom of the price continuum.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#146

Post by ludjer »

philliphaydon wrote: post #88
tiew9Vii wrote: post #74

Earlier quoted context omitted.

I hear that said a lot. I don’t think it stands. A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t. The cloud actually requires more investment for large organisations. Previously you might of only had a handful of sysadmins but now you have a large dedicated platform team doing devops type work building…

> A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t. Ani. If the hardware you’re running on is dying. In ec2 you stop it and start it. It’s on new hardware. If you run bare metal you’re screwed. Disk is dying ? Don’t matter cos your data exists multiple times over in AWS elastic storage. With bare metal y…

Not really If you running bare metal and have a San you can easily change what it is pointing to. Also most bare metal servers have redundancy and disks can be changed with 0 down time.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#147

Post by mywittyname »

vmception wrote: post #124
the_duke wrote: post #59

Earlier quoted context omitted.

In addition: long before AWS you could easily rent virtualized or dedicated servers. It took longer to provision than EC2 and all you had for storage is a fixed amount of disk, but that was absolutely sufficient for many businesses.

> but that was absolutely sufficient for many businesses. yes, but I am glad about how many job titles are obsoleted for many businesses due to how compute instances are managed. much smaller organizations used to need a full blown database administrator or two, and other personnel dedicated to keeping the server up. or you were doing it all yourself and spending your time on that. much higher barrier of entry than t…

Honestly, it didn't seem to be that different back then than it is now.

When I worked at smaller places, developers handled infrastructure and development. As the company grew, dedicated specialists came onboard to help.

Today, at smaller places, developers handle the cloud infrastructure, and as the company grows, they bring on dedicated specialists to help.

The biggest difference, I think, is that we have so many specialized products. We are no longer trying to figure out how to make a shoehorned relational DB scale, instead we start with a database designed for specific workloads.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#148

Post by simonebrunozzi »

the_duke wrote: post #59
manigandham wrote: post #9

> "Starting a web-based or SaaS (Software as a Service) business was virtually unheard of before the age of IaaS" Nonsense. There were plenty of SaaS startups. There was even a little event called the dotcom boom all about internet companies. This lack of history and experience is why new companies get into this cloud-first mess in the first place. Cloud is primarily for flexibility in iteration, dynamic scaling, or…

In addition: long before AWS you could easily rent virtualized or dedicated servers. It took longer to provision than EC2 and all you had for storage is a fixed amount of disk, but that was absolutely sufficient for many businesses.

> In addition: long before AWS you could easily rent virtualized or dedicated servers.

I disagree with this statement. Yes, you could rent, but not by the hour and based on compute power, and couldn't rent extra storage again by the hour and by the GB. Plus, you couldn't interact with these "virtual servers" through APIs.

I was at AWS 2008-2014 (early days!), and I think you should consider the impact of the "on-demand", API-by-default, nature of the AWS offering. Oh, and don't forget that with a valid credit card you could be up and running in literally minutes, not weeks.

Back then AWS had decent performance, but it was pretty bad when compared to more traditional Colo offerings; but in regards to the above aspects it dominated the scene, undisputed. IMHO, that's what gave AWS most of the initial traction.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#149

Post by simonebrunozzi »

SpaceMartini wrote: post #69

I work in HPC for a cloud provider, and fully endorse this move. Anonymously, of course. You can make an economic argument for or against cloud in practically every IT domain, but in HPC the case for on-prem is really compelling; none of the cloud networking/resiliency value-add is relevant to batch workflows, and costs per core-hour are only remotely comparable if you use spot - which is itself a major compromise. T…

> If I were independent I would recommend people buy and build on-prem clusters and shuffle data out of fast scratch into Glacier,

Pretty much 100% agree with this statement. Only exception is maybe at the beginning of a long series of computation, you might want to start on-demand to fully understand and size your exact needs, and then provision off-prem.

Re: We decided to move 90% of our workload from the cloud to on-prem infrastructure

#150

Post by gnarcoregrizz »

I always got the impression that there has been a lot of cloud propaganda/astroturfing, even on HN.

I'm seeing more of these "on-prem infrastructure" posts, citing costs, efficiency, and cloud complexity. We run most of our infra on-prem, and have looked to moving a few bits and pieces to the cloud, but the math almost always works out to buying hardware. Meanwhile, I talk to some friends, and their opex costs are astronomical for the traffic and size of their products.

The cloud is extremely convenient, and I would choose it if I were launching a new product, but past certain sizes and expenses I would start to do some math. It's not terribly difficult to run these cloud "shrinkwrapped" products (such as load balancers) on-prem. Things like object storage seem more difficult to me. I'm also hesitant to admin a database, they intimidate me :)

Post reply on HN