Live data from Hacker News

Convergence to Kubernetes

medium.com

111–120 of 124 posts

Re: Convergence to Kubernetes

#111

Dumb question from someone who doesn't use Kubernetes (or Docker) in production: don't routine security updates mean you're constantly rebuilding and redeploying these images? And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server?

One point that's not been addressed in your other replies is that Docker images should be a lot smaller than a "real" server so there should be fewer issues to patch. If you base off a very minimal image (and make use of things like multi-stage builds to remove dev. tools from the image) you can get the package numbers down a lot. For the ultimate in low dependencies of course you can build off scratch and put a sing…

> One point that's not been addressed in your other replies is that Docker images should be a lot smaller than a "real" server so there should be fewer issues to patch.

Could you quantify this?

It's possible to strip down a "real" server much farther than even what distros' "minimal" packages specify, which is why I wonder just how much smaller "a lot smaller" really is.

I commented on a short sub-thread [1] discussing dependency bloat/radii of distros. One commentor quantified it in bytes, though perhaps a more meaningful number in this context would be number of packages.

[1] https://news.ycombinator.com/item?id=17345982

Re: Convergence to Kubernetes

#112
post #88

Earlier quoted context omitted.

I don't think people primarily choose Docker because it's more efficient(?) to deploy than a real server. The benefits of Docker, to me, are having a single artifact and reproducible builds where a developer can run the docker image locally, it then gets built once on Jenkins, and the same image is deployed to staging, production, etc. It eliminates an entire class of problems related to the operating system, install…

docker containers are not reproducible. if you built the same content on machine a and built the same content on machine b the hash of the image would differ. and the timestamp probably too

Yes, but if you build the image and push it to a repository, everybody working on it can pull it and run it.

Re: Convergence to Kubernetes

#113

Earlier quoted context omitted.

> simple declarative YAML files If the config files and underlying infrastructure are changing all the time and have varying degrees of documentation, that's just shifting know-how from established tools to the newest fad, especially when k8s know-how is extremely scarce/expensive, and will leave you in a trial-and-error situation with unclear diagnostics if anything goes wrong due to the sheer complexity. Automation…

> k8s is just oversold and overhyped Yes, like anything else is, but it also removes almost all ops overhead and is much faster and easier to maintain at the container level than recreating images and redeploying VMs. Since the vast majority of startups just want to run apps and aren't doing any sophisticated lower-level infrastructure, trading terraform/chef/puppet for k8s yaml is a net win, especially since it also…

> easier to maintain at the container level than recreating images and redeploying VMs

I'm not sure I understand how that could be, since the best practice alluded to elsewhere in the thread is to recreate and redeploy (as part of ones normal CI/CD process, assuming that even exists) containers as a way of keeping up on security updates. How is that different, let alone easier, than doing so with VMs?

> startups just want to run apps and aren't doing any sophisticated lower-level infrastructure

That may be important at the very earliest stages, but that kind of concerted ignorance can be dangerous.

> it also replaces several other accessory software

The "also" being in addition to abstracting away all that tedious, "accessory" software. The problem is that, abstractions are leaky. At some point, it may well become important for someone in that startup to understand how/why the accessory works because it suddenly became critical to the business.

Of course, that point may never come, if the startup doesn't survive that long, so why bother thinking that far ahead?

Re: Convergence to Kubernetes

#114
post #15

> We have close to 30 teams that run some or all of their workloads on our clusters. Approximately 70% of all HTTP traffic we serve is generated from applications within our Kubernetes clusters. Sounds big. But then per wikipedia: > uSwitch.com [...] allows consumers to compare prices for a range of energy, personal finance, insurance and communications services. And: > On 30 April 2015, the property website firm Zoo…

> 2-3 hand-managed servers

I think too many people in this sub-thread are harping on this excerpt, especially with regard to the "hand-managed" and failing to take the most charitable reading, as exhorted by the guidelines.

What commenters are, perhaps, failing to grasp, is that if there are only 3 (or even 6) servers in the environment, and each one has a unique configuration, then they're all, by some definition, hand-managed.

No amount of "automation" will reduce that fundamental administrative/cognitive burden.

Sure, there are some sensible best practices, even for that scenario, like, at minimum, storing config in version control. However, going all-in on a CM system (or, perhaps, Kubernetes) for 100 servers when all you have is 3 is, at best, premature optimization. There's a well-known aphorism in tech about that.

Re: Convergence to Kubernetes

#115
post #114
post #15

> We have close to 30 teams that run some or all of their workloads on our clusters. Approximately 70% of all HTTP traffic we serve is generated from applications within our Kubernetes clusters. Sounds big. But then per wikipedia: > uSwitch.com [...] allows consumers to compare prices for a range of energy, personal finance, insurance and communications services. And: > On 30 April 2015, the property website firm Zoo…

> 2-3 hand-managed servers I think too many people in this sub-thread are harping on this excerpt, especially with regard to the "hand-managed" and failing to take the most charitable reading, as exhorted by the guidelines. What commenters are, perhaps, failing to grasp, is that if there are only 3 (or even 6) servers in the environment, and each one has a unique configuration, then they're all, by some definition, h…

Heh, yeah. I realized after the third reply that my phrasing had turned into a downvote magnet.

There's also the problem that the cluster itself becomes a failure point. It too needs to run on physical servers that need to be provisioned and managed. Anyone who can fat-finger a regular iron server configuration can muck up a kubernetes deployment.

Ultimately, the top of the stack is always "hand-managed" in some sense. Even AWS has had user-induced failures.

Re: Convergence to Kubernetes

#116
post #113

Earlier quoted context omitted.

> k8s is just oversold and overhyped Yes, like anything else is, but it also removes almost all ops overhead and is much faster and easier to maintain at the container level than recreating images and redeploying VMs. Since the vast majority of startups just want to run apps and aren't doing any sophisticated lower-level infrastructure, trading terraform/chef/puppet for k8s yaml is a net win, especially since it also…

> easier to maintain at the container level than recreating images and redeploying VMs I'm not sure I understand how that could be, since the best practice alluded to elsewhere in the thread is to recreate and redeploy (as part of ones normal CI/CD process, assuming that even exists) containers as a way of keeping up on security updates. How is that different, let alone easier, than doing so with VMs? > startups just…

Kubernetes is a solution for 90% of the effort and services needed for distributed apps running and interacting with each other over a cluster of servers. Those accessory services aren't abstracted, I never used that word. They are instead provided out of the box by K8S, leading to fewer individual components to run, maintain and monitor yourself. You can switch out components at any level, from a single container running your favorite webserver, to a full service mesh, to your own custom controllers.

Containers are much smaller than VMs, all the way down to just a single binary for your application if you want. Definition files are smaller. Restarting or replacing a container is much faster.

Your comment amounts to a big "what if", but this isn't a complex topic. Focusing on what you actually need is what leads to success. Any competent technical leadership will plan ahead and care about low-level detail when necessary. But you definitely don't need to unnecessarily worry about it, and that's what Kubernetes helps with.

Of course you can skip K8S altogether and have your setup, but that is rarely needed because it can now be solved with a common industry framework instead of bespoke solutions or special PaaS provider APIs. None of this is revolutionary, and no different than any other cost/benefit analysis for build/buy of any other component.

Re: Convergence to Kubernetes

#117
post #113

Earlier quoted context omitted.

> easier to maintain at the container level than recreating images and redeploying VMs I'm not sure I understand how that could be, since the best practice alluded to elsewhere in the thread is to recreate and redeploy (as part of ones normal CI/CD process, assuming that even exists) containers as a way of keeping up on security updates. How is that different, let alone easier, than doing so with VMs? > startups just…

Kubernetes is a solution for 90% of the effort and services needed for distributed apps running and interacting with each other over a cluster of servers. Those accessory services aren't abstracted, I never used that word. They are instead provided out of the box by K8S, leading to fewer individual components to run, maintain and monitor yourself. You can switch out components at any level, from a single container ru…

> They are instead provided out of the box by K8S, leading to fewer individual components to run, maintain and monitor yourself. You can switch out components at any level, from a single container running your favorite webserver, to a full service mesh, to your own custom controllers.

Fair enough, though I was misled by the word "replaces", which, to my mind, means something different than merely providing out of the box.

> Containers are much smaller than VMs

I don't disput that. I'm also well aware that size matters. However, you initial assertion was that it "removes almost all ops overhead" and that it is "much faster and easier to maintain" (emphasis mine), which merely increasing performance (as performed by computers, not humans), no matter the degree, doesn't do.

I'm still not seeing how the maintenance (done by humans) is any easier (or, for that matter, faster) than with VMs.

> Your comment amounts to a big "what if"

Only on the last sentence asks such a question. The rest of the comment asks different questions, some of which you've addressed above.

> Any competent technical leadership

This is a bit too "true Scotsman" to be useful. The question is what actually happens with actual leadership once a particular tool is in place. Does it encourage (inadvertently or otherwise) long-term dependence on the tool's ecosystem to the exclusion of those bespoke changes, or does it naturally peel away the leaky abstractions once they no longer hold value?

There are lessons to be learned from ORMs. Even early on, there was little (no?) controversy that the best practice would be to start off using them everywhere initially and replace them as needed. I don't believe this practice was followed, despite the pain, at least partially due to the perceptions of ease and consistency in sticking with ORM-everywhere.

> But you definitely don't need to unnecessarily worry about it

You may be conflating premature optimization (a strawman you detail in the next paragraph, which I won't address) and merely considerting the future, the "what if we succeed?" question, which is what I'm advocating.

I say it's quite necessary to at least think about the consequences of a tool choice in the context of eventual success, especially if the initial investment is high, but even if not.

All that said, I hope you've noticed, I'm not actually making any assertions about Kubernetes, just asking questions (and challenging assertions). I tend to take a position of skepticism with anything that seems to gain popularity through network effect, as it becomes extremely difficult to separate facts from hype, even in (especially in?) anecdotes.

Re: Convergence to Kubernetes

#118

Dumb question from someone who doesn't use Kubernetes (or Docker) in production: don't routine security updates mean you're constantly rebuilding and redeploying these images? And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server?

> don't routine security updates mean you're constantly rebuilding and redeploying these images?

Yes and no; yes, because you should be rebuilding upon upstream changes, but no, because there will be fewer upstream changes. You can reduce the amount of software that is baked into a Docker image, compared to a full traditional distribution, thus reducing your attack surface and reducing the frequency with which the image needs to be rebuilt.

> And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server?

"Real" servers are always (exception: NixOS) stateful, even if they are only used to host stateless services. Uninstalled packages leave behind old service users, buggy packages depend on specific user/group IDs that conflict in production but not where they were built, conflicting ports that are already bound. Designing for immutable infrastructure makes it easier to reason about state in production and therefore makes production easier to manage.

Re: Convergence to Kubernetes

#119

Earlier quoted context omitted.

Yeah with bare mesos I don't have to do any of that, I just accept resource offers and release them when I'm done. The scheduler decides how much it will scale, not mesos. We like to create a new framework for each instance of the job (so we can track slow runs, trial-run new versions), so we can have anywhere between 0 and 50 similar frameworks all running and vying for 100% of the cluster. Naturally, we have cluste…

It was kind of hard to understand what you were describing without reading the Mesos Architecture documentation[0], but I think I get it now. When you say "you" just accept resource offers, you mean the frameworks (in mesos terminology) you're deploying correct? Weirdly enough, Mesos resembles a system I was building in my head that I thought could compete with Kubernetes... Taking the resource supplier (agents in me…

I've been running a reasonably large kubernetes cluster (~600 cores) in production for the last 6 months or so. I've had to deal with hotspots, both IO, CPU and load (which could be either). Getting requests and limits dialed has been a bit of a challenge.

I worked with Mesos a bit last year and miss the sophistication of the scheduler. Granted, I never ran it at the same scale, but I felt like the approach was much easier to grok.

I've seen a few projects floating around meant help dynamically tune resource utilization in k8s. By this time next year, I imagine those will be fairly commonplace. I'm definitely looking forward to using descheduler:

https://github.com/kubernetes-incubator/descheduler

Re: Convergence to Kubernetes

#120
post #99
post #9

"The result was a system composed of many wavefronts of change: some systems were automated with Puppet, some with Terraform, some used ECS and others used straight EC2. In 2012 we were proud to have an architecture that could evolve so frequently, letting us experiment continually, discovering what worked and doing more of it. In 2017, however, we finally recognised that things had changed. AWS is significantly more…

> [...] this time, it is different. Literally a quote from my managers (not related to k8s or aws alone, though). They also made sure to repeat that phrase a few times so it becomes more believable.

"What's different this time?" is a classic question to ask yourself when evaluating tech choices. Sometimes the answer is that something does make it different this time, it's not just a rhetorical device to dismiss things that have been tried before.

Most things don't take off the first time they're tried. Think of all the current mainstream or trendy stuff, like virtualization, massively parallel coprocessors (now called GPUs), AI, multicore processors, etc. They all tried to come to market decades ago and had just niche success.

Post reply on HN