Rolling your own servers with Kubernetes • Hacker News Board

Re: Rolling your own servers with Kubernetes

#41

Post by rabi_penguin » Mon, May 06, 2019, 4:39 PM UTC

falcolas wrote: post #32

lima wrote: post #11

Earlier quoted context omitted.

Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything. It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale whe…

> spend weeks arguing with AWS support Yup. We've run into that repeatedly. The "we didn't notice anything on our side, please send more screenshots and logs" gets really old when working with "managed services". Network packet losses/truncations, EKS control plane failures, cloudformation stacks getting stuck in really wierd states, inconsistent cloudformation implementations for new and existing services, ENI weird…

Running into one of those exact issues (cloudformation stacks getting stuck in really weird states) right now. A real pain in the ass, because I have zero control or visibility into it.

Re: Rolling your own servers with Kubernetes

#42

Post by acd » Mon, May 06, 2019, 4:46 PM UTC

There is also kubespray Ansible provisioning of Kubernetes, kops and kubeadm.

Rolling Kubernetes on your own is quite hard though especially the networking part so there should be a market for a company helping out with provisioning.

Using public cloud can be very expensive for some work loads which were not written for the cloud due to higher resource usage. It is still quite hard to forecast cost of the cloud due to a lot of moving parts and micro billing for each item.

With traditional service providers it can be easier to budget for the service expenses.

Re: Rolling your own servers with Kubernetes

#43

Post by isatty » Mon, May 06, 2019, 4:48 PM UTC

falcolas wrote: post #32

lima wrote: post #11

Earlier quoted context omitted.

Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything. It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale whe…

> spend weeks arguing with AWS support Yup. We've run into that repeatedly. The "we didn't notice anything on our side, please send more screenshots and logs" gets really old when working with "managed services". Network packet losses/truncations, EKS control plane failures, cloudformation stacks getting stuck in really wierd states, inconsistent cloudformation implementations for new and existing services, ENI weird…

Cloudformation is a tragedy in itself.

Re: Rolling your own servers with Kubernetes

#44

Post by nocman » Mon, May 06, 2019, 4:55 PM UTC

nemonemo wrote: post #22

lima wrote: post #18

Earlier quoted context omitted.

How is that related to bare metal vs. clouds? That could have happened on either.

Big cloud providers or some big VC backed companies have the resource to figure out the issues by wrestling with the gigantic and unyielding vendors, or by working around it with their engineering resource. Not many teams have the time for even wanting to understand the issues. This is not an inherent problem of bare metal, but one aspect to consider if you were to choose the path. But you are right, cloud providers…

> Big cloud providers or some big VC backed companies have the resource to figure out the issues by wrestling with the gigantic and unyielding vendors, or by working around it with their engineering resource.

This is all operating on the assumption that those entities will care enough about your problem enough to do something meaningful to fix it. As others have stated here, at least in the context of "Big cloud providers", that frequently is not the case. Often you just get the runaround (continuous delays, requests for "more information", other attempts to stall in first level support).

Add a third party "managing" some piece of your company's infrastructure via a cloud provider into the mix and it often gets even worse.

It is true that there are real costs to having your own hardware/software onsite and people who know how to manage it. However, the promised reductions in cost/hassle of moving things offsite are frequently offset or even exceeded by the costs/hassles you get by not having control of things yourself.

Re: Rolling your own servers with Kubernetes

#45

Post by djsumdog » Mon, May 06, 2019, 4:58 PM UTC

acd wrote: post #42

There is also kubespray Ansible provisioning of Kubernetes, kops and kubeadm. Rolling Kubernetes on your own is quite hard though especially the networking part so there should be a market for a company helping out with provisioning. Using public cloud can be very expensive for some work loads which were not written for the cloud due to higher resource usage. It is still quite hard to forecast cost of the cloud due t…

I did a post on installing DC/OS a few years back. I was going to follow up with a K8s post, but between the official Ansible scripts, the (at the time) very alpha kubeadm, broken pod logging and WeaveNet that would constantly fail with no logs, I ended up giving up on that post:

https://penguindreams.org/blog/installing-mesosphere-dcos-on...

I remember the manual-ish server provisioning days of Autoyast and RedHat's Kickstart. It's not that difficult to bootstrap physical servers to get them to the point of being nodes in a Docker scheduling system. Rancher is another great tool for setting up K8s (which you can run against physical nodes once you get them past the initial install/bootstrap).

I worked at one place where they took a hybrid approach. They had DC/OS running in the local VMWare data center (which we were migrating to from AWS to save costs), but there were nodes that could scale to AWS as well and you could label your deployments for which data center you needed them to run in. A lot of high volume loads we were able to move back to our data center, but we also had a kick-ass platform team and our AWS bill was well over $150k/month.

The trouble is it's almost impossible to do a real cost comparison. There are just too many factors, and when you start going AWS, GCE or Azure, you start using all their other managed services that may not have self-hosted replacements, not to mention off-setting the costs of setting up master/slaves, backups, snapshots, redundancy and other operations tasks that come by default or with an additional price from a hosted solution.

Re: Rolling your own servers with Kubernetes

#46

Post by Nullabillity » Mon, May 06, 2019, 5:01 PM UTC

nemonemo wrote: post #22

lima wrote: post #18

Earlier quoted context omitted.

How is that related to bare metal vs. clouds? That could have happened on either.

Big cloud providers or some big VC backed companies have the resource to figure out the issues by wrestling with the gigantic and unyielding vendors, or by working around it with their engineering resource. Not many teams have the time for even wanting to understand the issues. This is not an inherent problem of bare metal, but one aspect to consider if you were to choose the path. But you are right, cloud providers…

Cloud providers also tend to run much more exotic configs for pretty much everything (since tenant isolation is a top priority), so such issues are more likely to happen in the first place.

Re: Rolling your own servers with Kubernetes

#47

Post by turk73 » Mon, May 06, 2019, 5:01 PM UTC

lima wrote: post #9

NickBusey wrote: post #6

I am curious to see how they recommend handling storage on bare metal k8s.

In many environments, you don't actually need distributed storage: - Application servers are stateless. - All state lives in stateful databases. - Those database handle high availability at the application layer by replication data, so you can use local storage and deploy them using StatefulSets. Has the extra benefit of avoiding the network storage latency penalty.

But StatefulSets require storage...someplace.

We're dealing with that exact issue right now. Our K8s cluster is not playing nice with a NetApp NAS. We are exploring other options such as local iscsi storage volumes ($$$) or an external state store db (slow).

Re: Rolling your own servers with Kubernetes

#48

Post by ljm » Mon, May 06, 2019, 5:02 PM UTC

acd wrote: post #42

There is also kubespray Ansible provisioning of Kubernetes, kops and kubeadm. Rolling Kubernetes on your own is quite hard though especially the networking part so there should be a market for a company helping out with provisioning. Using public cloud can be very expensive for some work loads which were not written for the cloud due to higher resource usage. It is still quite hard to forecast cost of the cloud due t…

From my limited experience, you'd save a lot on the cloud if you replaced the default IngressController. Google Cloud, for instance, will spin up a fresh load balancer for every service by default and those cost money.

Of course, it's not an easy task as a beginner to use nginx or traefik in its place, and to handle the complexity of that deploy.

Re: Rolling your own servers with Kubernetes

#49

Post by toomuchtodo » Mon, May 06, 2019, 5:02 PM UTC

rabi_penguin wrote: post #41

falcolas wrote: post #32

Earlier quoted context omitted.

> spend weeks arguing with AWS support Yup. We've run into that repeatedly. The "we didn't notice anything on our side, please send more screenshots and logs" gets really old when working with "managed services". Network packet losses/truncations, EKS control plane failures, cloudformation stacks getting stuck in really wierd states, inconsistent cloudformation implementations for new and existing services, ENI weird…

Running into one of those exact issues (cloudformation stacks getting stuck in really weird states) right now. A real pain in the ass, because I have zero control or visibility into it.

Is switching to Terraform an option?

Re: Rolling your own servers with Kubernetes

#50

Post by [deleted] » Mon, May 06, 2019, 5:03 PM UTC

falcolas wrote: post #34

closeparen wrote: post #8

Earlier quoted context omitted.

At the scale where owning infrastructure makes sense, that's not a distraction but some team's full-time mission.

Even before that tipping point, you're likely to have a team dedicated to working with infrastructure hosted by a cloud provider. Based off past (hardware) and current (AWS) experience, the team sizes aren't that different.

[deleted]