Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

srcco.de

Post reply on HN
Print view View original

81–90 of 242 posts

Re: Kubernetes Failure Stories

#81

Post by jordanbeiber »

romeisendcoming wrote: post #75
jordanbeiber wrote: post #63

Earlier quoted context omitted.

In my experience most companies lack common conventions and automations. Kubernetes "done right" is almost a part of your application. It becomes this "machine" that you throw stuff into and good stuff happens. You'll need a team to integrate it into the pieces you require (auth, secrets, loadbalancers, permissions/app identities, monitoring and logging) but many places lack bits and pieces, and in my opinion k8s giv…

Those are your startups and web/app tier shops. Yes, they suck at sysadmin routinely and they need to be bottle fed a solution that fits the scatter/gather shape of their business. They don't want OPs discipline. They want a programmable solution that performs systems magic with a single toolset to learn.

No no, these are your enterprises I’m talking about mainly.

Places entrenched in manual processes for release and change management.

With true service delivery in a CI/CD fashion (including infrastructure, as it should be codified) many of these manual processes becomes obsolete.

Don’t get me wrong, the processes still exist, they are just sped up by a magnitude and automated.

Re: Kubernetes Failure Stories

#82

Post by dilyevsky »

wikibob wrote: post #73
dilyevsky wrote: post #69

Earlier quoted context omitted.

Re configuration: ksonnet is an option (although I personally find jsonnet a “lipstick on a pig” kind of solution). There’s some work going on to have something more user-friendly (think Google’s Piccolo) - https://github.com/stripe/skycfg (disclaimer - I contributed to this project)

Could you describe Piccolo a bit? Can't find anything on it.

Not going into too much details- it was a Python-esque dsl equivalent to bcl. You still had to learn Borg abstractions but at least you didn’t have to fight the language as much if you wanted to implement DRY in your configs.

Re: Kubernetes Failure Stories

#83

Post by romeisendcoming »

parasubvert wrote: post #46
bdcravens wrote: post #34

I've started the planning phase of a Kubernetes course, geared toward developers more so than the enterprise gatekeepers. As I read stories like these, I jump between different thoughts and feelings: 1) no matter what I think I know, there's too many dark corners to create an adequate course 2) K8S is such a dumpster fire that I shouldn't encourage others 3) there's a hell of an opportunity here Thoughts? Worth pursu…

All three. It’s a gold rush, but as with any gold rush, conditions are hard going - that’s why there’s an opportunity. Best way to think of Kubernetes is that it was designed to be a successful open source project that was widely adopted as a standard foundation to build products. It wasn’t designed to be a useable product on its own. We are at the equivalent stage of Slackware and SLS and Debian Red Hat pre-1.0 stag…

Don't forget SuSE the sole surviving competitor. Best Buy SuSE Linux gecko box 2.2.14 kernel veteran.

Re: Kubernetes Failure Stories

#84

Post by dcomp »

I run a single node cluster at home. In order to handle updates. I just wipe the cluster with kubeadm reset. Then kubeadm init; followed by running a simple bash script. which loops of files in nested subdirectories applying yaml configs. Only have to make sure I only ever edit the yaml files and not mess with kubectl edit etc.

for f in /.yaml ...

with a directory structure of:

  drwxrwsrwx+ 1 root 1002 176 Jan 20 21:15 .
  drwxrwsrwx+ 1 root 1002 194 Nov 17 20:06 ..
  drwxrwsrwx+ 1 root 1002  68 Jan 20 20:50 0-pod-network
  drwxrwsrwx+ 1 root 1002 104 Nov  1 11:18 1-cert-manager
  drwxrwsrwx+ 1 root 1002  34 Jul 11  2018 2-ingress
  -rwxrwxrwx+ 1 root 1002  93 Jan 20 21:15 apply-config.sh
  drwxrwsrwx+ 1 root 1002  22 Jul 14  2018 cockpit
  drwxrwsrwx+ 1 root 1002  36 Jul  3  2018 samba
  drwxrwsrwx+ 1 root 1002  76 Jul  6  2018 staticfiles

Re: Kubernetes Failure Stories

#85

Post by romeisendcoming »

jordanbeiber wrote: post #81
romeisendcoming wrote: post #75

Earlier quoted context omitted.

Those are your startups and web/app tier shops. Yes, they suck at sysadmin routinely and they need to be bottle fed a solution that fits the scatter/gather shape of their business. They don't want OPs discipline. They want a programmable solution that performs systems magic with a single toolset to learn.

No no, these are your enterprises I’m talking about mainly. Places entrenched in manual processes for release and change management. With true service delivery in a CI/CD fashion (including infrastructure, as it should be codified) many of these manual processes becomes obsolete. Don’t get me wrong, the processes still exist, they are just sped up by a magnitude and automated.

Who said anything about manual processes? That's not what the modern SA does...it's mostly designing repeatable processes, creating recipes and integration in my experience.

The real problem with the K8s and devops world is no understanding of why there is no magic pill in 'codifying' a bad system.

Re: Kubernetes Failure Stories

#86

Post by cygned »

jacques_chester wrote: post #79
cygned wrote: post #49

I am a developer and I find k8s frustrating. To me, its documentation is confusing and scattered among too many places (best example: overlay networks). I have read multiple books and gazillions of articles and yet I have the feeling that I am lacking the bigger picture. I was able to set it up successfully a couple of times, with more or less time required. Last time, I gave up after four days because I realized tha…

Kubernetes has always had an identity crisis. Who is aimed at, app developers or platform operators? Clear, obvious contracts between the two roles are valuable, even if you decide to combine them. I'm moderately hopeful that Knative will help in that regard, as it is more conclusively oriented towards the developer. But I am wary that since it leaves the implementation details completely visible, it may not achieve…

Achieving that is a difficulty task, though. Personally, I’d not like to rely on an abstraction on top of a system with that level of complexity for production because I expect to run into situations that can only be solved with deep knowledge of k8s.

Re: Kubernetes Failure Stories

#87

Post by lmm »

jordanbeiber wrote: post #56
stingraycharles wrote: post #28

Earlier quoted context omitted.

The biggest reason for choosing microservices _should_ be scaling development teams: microservices allow multiple teams to work on different code bases, without stepping on each other's toes. What actually happens, tough, is that (uninformed) people choose it because they think it brings them scalability (wrong), it's more cloud compatible (wrong) or the worst offender, it's more modern.

If you achieve this scale of teams suggested, it means you have a lot of implicit gains here as well, right? Such as higher quality, higher "velocity", separation of concern and hopefully a clear sense of ownership. These are some of the things in my opinion that allows you to scale.

> If you achieve this scale of teams suggested, it means you have a lot of implicit gains here as well, right?

> Such as higher quality, higher "velocity", separation of concern and hopefully a clear sense of ownership.

No, it means you pay a huge overhead. Quality and velocity both drop as your day-to-day development requires a lot more setup and faff to do anything, and counterintuitively so does separation of concerns as your interfaces become more rigid. Small organisations should do things that don't scale, turn their size into an advantage.

If you think of your overhead as ax + bx^2 where x is the number of developers, microservices are a way to reduce b, but at the cost of a big increase to a. It makes sense when x is huge but not before. My litmus test would be: do you need to do multiple (unrelated) deployments of different services at once? If your organisation is small enough that you can get away with only deploying one thing at a time, you'll probably have less overhead if you work without microservices.

Re: Kubernetes Failure Stories

#88

Post by jordanbeiber »

romeisendcoming wrote: post #85
jordanbeiber wrote: post #81

Earlier quoted context omitted.

No no, these are your enterprises I’m talking about mainly. Places entrenched in manual processes for release and change management. With true service delivery in a CI/CD fashion (including infrastructure, as it should be codified) many of these manual processes becomes obsolete. Don’t get me wrong, the processes still exist, they are just sped up by a magnitude and automated.

Who said anything about manual processes? That's not what the modern SA does...it's mostly designing repeatable processes, creating recipes and integration in my experience. The real problem with the K8s and devops world is no understanding of why there is no magic pill in 'codifying' a bad system.

I did. As I see it all to often mainly at the larger places.

Modern SA is about knowing that your job to help bring business value. Most of the time this is down to automation.

Re: Kubernetes Failure Stories

#89

Post by 9dev »

jordanbeiber wrote: post #37
lucidone wrote: post #18

Earlier quoted context omitted.

I'm consulting on a micro services back end right now with mostly prior experience with monoliths. What is the selling point that drives companies down this direction? It's insane, and my client keeps trying to hire new developers and bring on more consultants to build this thing, but the amount of knowledge required is more than any one person can handle. I have similar issues with their choice of db (nosql) and its…

Microservices requires proper, structured systems management. Most seem to think "devops" and microservices are ways to ignore this, when in fact the complete opposite applies. Make sure you have a true service delivery and service management pipeline in place where it is real easy for dev and ops to deploy and decommission services. Service metadata is key in my experience. Not too much though, just enough (such as…

But why!?

Re: Kubernetes Failure Stories

#90

Post by m0zg »

trhway wrote: post #64
m0zg wrote: post #51

It's not for everyone and it has significant maintenance overhead if you want to keep it up to date _and_ can't re-create the cluster with a new version every time. This is something most people at Google are completely insulated from in the case of Borg, because SRE's make infrastructure "just work". I wish there was something drastically simpler. I don't need three dozen persistent volume providers, or the ability…

An open source having complexity of an enterprise monster - this is what generates the half million plus salaries. An old enterprise software trick. Simplification of it would serve no interests of anybody in the position to do the simplification.

I'm going to voice a contrarian viewpoint here and say that "half million plus" salaries are actually a good thing. Rising tide lifts all boats and since a lot of technies live in areas with exorbitant cost of living, that money re-enters the economy at a rapid clip anyway. But such salaries are only good if commensurate value is being delivered for the money. Which in a large IT shop it might be, but as a small business owner K8S is a hard slog, hence my suggestion to simplify. I'm pretty sure 80/20 breakdown still applies, and 80% of K8S complexity could be removed without affecting anything much. One might suggest that I use GKE and bypass the problem entirely, but I need to run a lot of GPUs 24x7, and the pricing on those in any cloud is insane.
Post reply on HN