Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is K8s Too Complicated? 

jmoiron.net

Post reply on HN
Print view View original

161–170 of 171 posts

Re: Is K8s Too Complicated? 

#161

Post by mwcampbell »

jacques_chester wrote: post #113
matthewmacleod wrote: post #57

I suppose we have to assume that we should use the right tool for the right job, and all that. And I'm sure that the Kubernetes folk know what they're doing. But I definitely think that it's too complicated without a cutting-edge Kubernetes expert in place to manage it. And even then, it's just a building block for a larger system. I've tried maybe half a dozen times to get started for relatively small workloads – le…

> I'd love something like Heroku that I could run on my own bare metal, or on AWS – the various solutions I've tried were all lacking. I've worked on or near to Cloud Foundry for about 4 years now, and Heroku was one of the inspirations for it, along with Borg. Is there something in particular it didn't do well enough?

I'm not the GP. But if I were setting up infrastructure for a fledgling startup today, I'd want a PaaS-in-a-box that worked like this:

Given these inputs:

* A minimum of 3 Linux servers: These could be VMs or bare-metal servers. They're all in the same data center. They each have 1 public IP address. Other than that, there's no network between them. Each of these nodes has at least 4 GB of RAM and a healthy amount of SSD storage. Note that I'm perfectly comfortable manually provisioning a fixed number of nodes, and manually adding more if truly necessary, because predictable cost is important.

Edit: The platform should assume that each node has a bare installation of Ubuntu or CentOS. Don't make me install something custom. And ideally, don't be picky about the kernel, because some dedicated server hosts provide their own.

Edit 2: All nodes should be treated equally. Any of them may become the master when needed. There should be no dedicated master; all nodes should be available to run applications. After all, resources are tight, and I want to get the most out of those three servers.

* An API for creating and updating DNS records, e.g. Amazon Route 53, DNSimple, etc.

* Edit: An off-site object store (e.g. Amazon S3, Backblaze B2) where the cluster can automatically send backups of all durable storage.

The PaaS-in-a-box should give me an installation script to run on each node. During installation, I provide my DNS API credentials and the domain(s) I want the platform to manage. Edit: I'd also provide credentials for the off-site backup object storage. And I forgot that when installing on the second and subsequent nodes, I'd provide the public IP of an existing node during installation.

And that's it. The cluster then manages itself, distributing durable storage among the nodes using something like Ceph for file storage, and DB-level replication for the supported database(s). HTTP/HTTPS traffic can be load-balanced among nodes using round-robin DNS, with unhealthy nodes automatically removed from the DNS records by one of the remaining nodes. If I need to run an outward-facing non-HTTP service, I should be able to reserve one or more ports for it and run it in either a round-robin or active/standby configuration with automatic failover.

Perhaps this is a tall order. Is it impossible to do this on top of something as last-decade as a handful of manually provisioned dedicated servers, with only the public network between them (but in the same data center)? I hope not, because that kind of server, while now out of fashion, is attractive to a company on a shoestring budget that nevertheless doesn't want to compromise performance.

Anyway, if a product meeting these requirements exists, I don't yet know about it. Cloud Foundry certainly doesn't market itself for that kind of deployment.

Re: Is K8s Too Complicated? 

#163

Post by bavell »

barnabee wrote: post #90
013a wrote: post #34

"In a single tweet — can you name a technical benefit you and your team have gained by switching to Kubernetes?" "The missing step in a comprehensive OSS devops strategy, from code (git) build (Docker) test (Docker) to deploy (Kube), which enables push-button CICD like no one has gotten right since Heroku."

If they'd got it as right as Heroku then deployment (or CD configuration) would be a one liner once a server is set up and scaling etc. would be configurable via a very simple web console or CLI. Unless I missed something, last time we tried k8s (admittedly ~6 months ago) it was woefully far from that goal.

Although it didn't come out of the box, my deploys are now one-liners. The command builds, tests and packages my app into a docker image, then bumps the version number on the k8s deployment which triggers a rolling update of production pods. I'm very happy with my current deployment flow, mostly because it works reliability and I can forget about what it's doing under the hood.

Re: Is K8s Too Complicated? 

#164

Post by jacques_chester »

ec109685 wrote: post #152
jacques_chester wrote: post #118

Earlier quoted context omitted.

My advice is to go directly to a PaaS. I work for Pivotal R&D in and around Cloud Foundry, so that's my personal horse. But I'd rather that you used any PaaS -- Cloud Foundry, OpenShift, Rancher -- than roll your own. Building a platform is hard . It's really really hard. Kubernetes commoditises some of the hard bits. The community around it will progressively commoditise other aspects in time. But PaaSes already exi…

Building PaaS abstractions on top of Kubernetes is an order of magnitude easier than doing what you guys did with Cloud Foundry. Building something that can scale to 250k Containers is monumentally hard, but with K8s, it is taken care of for you: https://kubernetes.io/docs/admin/cluster-large/ If you are a large enough organization, it is quite feasible to setup Kubernetes, chose an ingress solution and then build te…

> Building PaaS abstractions on top of Kubernetes is an order of magnitude easier than doing what you guys did with Cloud Foundry

It's worth noting Kubernetes didn't exist when Cloud Foundry started. Neither did Docker. The reason Cloud Foundry built two generations of container orchestration technology (DEA/Warden and Diego/Garden) was because it was partly inspired by direct experiences of Borg, as Kubernetes was. Folks had seen the future and decided to introduce everyone else to it.

The point here is not whether sufficiently large organisations are able to build their own PaaSes. They absolutely can. Pivotal's customer roster is full of companies whose engineering organisations absolutely dwarf our own.

The question is: should you build your own? This is not a new question. Should I build my own OS? My own language? My own database? My own web framework? My own network protocol? My own logging system? My own ORM?

The general answer is: no, not really. It's not the most effective use of your time, even if it's something you'd be perfectly able to achieve.

Re: Is K8s Too Complicated? 

#165

Post by jacques_chester »

mwcampbell wrote: post #161
jacques_chester wrote: post #113

Earlier quoted context omitted.

> I'd love something like Heroku that I could run on my own bare metal, or on AWS – the various solutions I've tried were all lacking. I've worked on or near to Cloud Foundry for about 4 years now, and Heroku was one of the inspirations for it, along with Borg. Is there something in particular it didn't do well enough?

I'm not the GP. But if I were setting up infrastructure for a fledgling startup today, I'd want a PaaS-in-a-box that worked like this: Given these inputs: * A minimum of 3 Linux servers: These could be VMs or bare-metal servers. They're all in the same data center. They each have 1 public IP address. Other than that, there's no network between them. Each of these nodes has at least 4 GB of RAM and a healthy amount of…

> Cloud Foundry certainly doesn't market itself for that kind of deployment.

Historically customers of Pivotal, IBM, etc focused on high availability requirements, which requires more machines. I've seen the default deployment for megacorps and it is, essentially, "bring all your dollars". But that is what they want -- no single points of failure. That means multiple VMs for everything. In fact it usually means multiple AZs for everything -- every component replicated thrice in at least two widely separated locations.

Even so, we've done work to split the difference. There's a "small footprint" version of PCF which is 4-9 VMs depending on how much risk you feel like taking. There's also cfdev if you just want to kick tires.

For the rest I can point to this and that. Service brokers for stuff like the DNS, traffic is directed by Gorouter or TCPRouter currently (with plans to switch to Istio), backups by BOSH Backup & Restore (BBR), running on raw hardware if the provider uses RackHD or can give you a BOSH Cloud Programming Interface (CPI).

BOSH is probably where you'd need to make your deepest peace. It has a very emphatic model of operations, which is that you are building a distributed system from known-good states, so individual machines are there solely to be paved whenever necessary. BOSH manages everything down to the operating system on the machine. You give it keys to an API that can provision compute, disks and networks and it will do the rest.

The uniform node thing would be tricky and my hunch is that, requiring a bunch of de novo engineering, would be less reliable on average than current arrangements.

Re: Is K8s Too Complicated? 

#166

Post by NTDF9 »

tejohnso wrote: post #60

"I suspect that a significant source of programmers flooding into management is the decreasing wavelength of full scale re-education on how to accomplish the exact same thing slightly differently with diminshing returns on tangible benefits." Wow. I'm curious to see how others are interpreting this sentence. I've got: I suspect many developers opt to move on to management largely because of frustration with having to…

I agree. Most of software engineering has turned into resume driven development, and engineers can only keep up at it for some time.

Kubernetes did not have to be so complicated. Docker swarm just proved it. If it takes 10 engineers to really program and manage your kubernetes instances, what problem did it solve again?

Re: Is K8s Too Complicated? 

#167

Post by ec109685 »

jacques_chester wrote: post #164
ec109685 wrote: post #152

Earlier quoted context omitted.

Building PaaS abstractions on top of Kubernetes is an order of magnitude easier than doing what you guys did with Cloud Foundry. Building something that can scale to 250k Containers is monumentally hard, but with K8s, it is taken care of for you: https://kubernetes.io/docs/admin/cluster-large/ If you are a large enough organization, it is quite feasible to setup Kubernetes, chose an ingress solution and then build te…

> Building PaaS abstractions on top of Kubernetes is an order of magnitude easier than doing what you guys did with Cloud Foundry It's worth noting Kubernetes didn't exist when Cloud Foundry started. Neither did Docker. The reason Cloud Foundry built two generations of container orchestration technology (DEA/Warden and Diego/Garden) was because it was partly inspired by direct experiences of Borg, as Kubernetes was.…

I know Kubernetes wasn’t around when Cloud Foundry was started. That wasn’t my point. Some of your argument was that building Cloud Foundry was hard (and I agree!), therefore you need a vendor’s PaaS. That isn’t true.

If an engineering organization takes Kubernetes and adds their own tooling around it to turn it into a PaaS for their org, that isn’t in the same league as building their own Database or what you did with Cloud Foundry originally.

Re: Is K8s Too Complicated? 

#168

Post by jacques_chester »

ec109685 wrote: post #167
jacques_chester wrote: post #164

Earlier quoted context omitted.

> Building PaaS abstractions on top of Kubernetes is an order of magnitude easier than doing what you guys did with Cloud Foundry It's worth noting Kubernetes didn't exist when Cloud Foundry started. Neither did Docker. The reason Cloud Foundry built two generations of container orchestration technology (DEA/Warden and Diego/Garden) was because it was partly inspired by direct experiences of Borg, as Kubernetes was.…

I know Kubernetes wasn’t around when Cloud Foundry was started. That wasn’t my point. Some of your argument was that building Cloud Foundry was hard (and I agree!), therefore you need a vendor’s PaaS. That isn’t true. If an engineering organization takes Kubernetes and adds their own tooling around it to turn it into a PaaS for their org, that isn’t in the same league as building their own Database or what you did wi…

I brought up the history as a lot of people don't know it.

I think we've made our respective points.

And hey, for folks building tooling around Kubernetes, I'm pretty much obliged to mention PKS and Concourse :)

Re: Is K8s Too Complicated? 

#169

Post by SEJeff »

barrkel wrote: post #145
SEJeff wrote: post #112

Earlier quoted context omitted.

Configuration management absolutely can do this. Note that I'm a builder of kubernetes clusters (on-premise) but also have contributed 400+ patches to the salt configuration management tool. Salt has an "orchestrate layer"[1] which allows running states on sets of minions. One of those layers can be to configure a service, and another can be to update the load balancers when said service is healthchecking green. Sayi…

It's not about skill and approachability; it's about not having tens of thousands of systems in different companies built out of hacks and science experiments, where experience isn't portable and every system's hacks and idiosyncrasies need to be learned anew. K8s raises the abstraction level on deployment of a multi-node application architecture. Further, as it builds out standard abstractions for external services,…

Yes, I'm aware and you're preaching to the choir. I'm literally the lead on $employer's k8s deployment. I was just pointing out that things like k8s can absolutely be done reliably with old school bare metal and config management + some orchestration glue. Only it isn't as simple for most and generally is done poorly. I'm a huge fan of k8s.

Re: Is K8s Too Complicated? 

#170

Post by inteleng »

StavrosK wrote: post #136
vectorpush wrote: post #111

Earlier quoted context omitted.

The words have meaning. If you don't understand them you could just ask for them to be explained rather than throwing out insults. I suspect you actually do understand the meaning of what was written though so I suppose that means you're just trying to start a flamewar.

The problem is that "leveraging" can be replaced with "using" every time, with the added benefit that it won't leave a bad taste in the reader's mouth.

This is what I mean.
Post reply on HN