Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

srcco.de

Post reply on HN
Print view View original

191–200 of 242 posts

Re: Kubernetes Failure Stories

#191

Post by koffiezet »

stunt wrote: post #50

Kubernetes solves a problem that most of the companies don't have. That is why I don't understand why the hype around it is so big. For the majority, it just adds a little value when you compare to added complexity to infrastructure and the cost of a learning curve and the ongoing operation and maintenance.

Don't agree. My current client is rather small, but I just had a meeting which would have been concluded with "we'll have to set up 15 vm's before you can start developing and have 5 alignment meetings before they're correctly set up", versus "I just created a namespace for you guys to do whatever you want in, and mailed you the link to the docs for the CI/CD and deploy guidelines".

They run a self-hosted OpenShift cluster, which is managed internally by a team of 4. Not only makes this situation it a lot easier to spin up new environments, it also forces devs to include the ops team from the start for stuff they don't know, and corrections can be made early on.

Re: Kubernetes Failure Stories

#192

Post by hjacobs »

zerogvt wrote: post #184
rawoke083600 wrote: post #182

Earlier quoted context omitted.

Very true !!! There is no greater culprit responsible for "complex systems" than the act of "extensible/future proof" in software design !

I think that the unix philosophy of focused and relatively simple tools that are easy to glue together is a better way to future-proof. Yet to do that you need to have a stable substrata to provide the basis of composition. In k8s case it seems that k8s _is_ the basis where the composition is to happen upon.

In conversations, I often compare the Kubernetes API to the Linux Kernel API (as analogy) - both provide primitives we kind of "agreed on" in the industry. I hope the Kubernetes ecosystem will flourish in the same way as the Linux base.

Re: Kubernetes Failure Stories

#193

Post by jacques_chester »

akhilcacharya wrote: post #163
bpicolo wrote: post #101

Earlier quoted context omitted.

> app developers or platform operators Definitely not the former. The YAML-based configuration is not a pleasant app deployment experience. Companies end up needing to do some sort of auto-generation for it to make it sane for app devs. App developers want experiences similar to heroku. They want to git push and have applications safely roll out without downtime or configuration.

I’m a little behind on Cloud Native adoption so I gotta ask - what’s preventing a “git push to heroku” from being the norm here? Are we using the wrong abstractions? Or are the abstractions still too low? ...why is Heroku/buildpack not running away with it?

The main alternative was and still is Cloud Foundry (by way of disclosure, I've been in and around CF for years). But it's always been pitched to enterprise customers and approximately zero effort has ever been made to expand awareness outside that world. Kubernetes had Google's massively powerful halo blowing in its sails.

> ...why is Heroku/buildpack not running away with it?

I predict that buildpacks are going to make a big comeback, because they make everything about Day 2 much easier. Pivotal and Heroku have been cooperating on the Cloud Native Buildpacks[0] spec under the CNCF sandbox process. There is some seriously cool stuff coming down the pipe.

[0] https://buildpacks.io/

Re: Kubernetes Failure Stories

#194

Post by ithkuil »

justicezyx wrote: post #171
ithkuil wrote: post #168

Earlier quoted context omitted.

I found https://jsonnet.org/ to fix several issues with the Borg configuration language. https://github.com/ksonnet/kubecfg is an attempt to reboot the borgcfg experience with k8s + jsonnet

There is no such thing as borgcfg experience. It’s just a configuration dsl applied to producing borg specs.

You might take it for granted, and thus not experience it as an "experience", but if you use other tools that are popular in the k8s world (such as helm) you might feel a tinge of nostalgia.

For example, {borg,kube}cfg allow you to import an existing config and override it so you can adapt it to another scenario (different things in different clusters, like prod vs staging, or a cluster has a new feature while another one doesn't etc).

Furthermore, the overrides are described with the same "shape" as the things they override, and can override things that weren't necessarily marked as overridable.

Compare this with the current state of affairs with helm, where the only way for users to be able to inject e.g. the resources requests and limits in a pod spec is for the original template author to have foreseen that need and explicitly added a hook in the template so that values from another file (the values.yaml) can be injected into it.

        spec:
          imagePullSecrets:
            - name: {{ .Values.image.pullSecret }}
          containers:
            - name: {{ template "mycontainer.name" . }}
              ......
              resources:
    {{ toYaml .Values.resources | indent 12 }}
          volumes:
              ......

Re: Kubernetes Failure Stories

#195

Post by bvm »

zerogvt wrote: post #183
m0zg wrote: post #51

It's not for everyone and it has significant maintenance overhead if you want to keep it up to date _and_ can't re-create the cluster with a new version every time. This is something most people at Google are completely insulated from in the case of Borg, because SRE's make infrastructure "just work". I wish there was something drastically simpler. I don't need three dozen persistent volume providers, or the ability…

I think Istio ( https://istio.io ) is a nice effort to create both an abstraction on top of k8s and to package a set of commonly needed functionality out of the box. Unsure of its production status or overhead though. Also I'd only go with a managed k8s solution and I'm not sure I'd consider k8s for older or non-microservice/containerized architectures. In the later case though I don't think there's anything better o…

Lots of magic for me, I've broken a (dev) k8s cluster by installing Istio via Gitlab k8s integration. The overhead appeared to be non-negligible, but I noped-out-of-there pretty quick, so I don't have the data to back that up.

Re: Kubernetes Failure Stories

#196

Post by devthane »

Aeolun wrote: post #155
devthane wrote: post #122

Earlier quoted context omitted.

This isn’t my experience at all. I as one person taught myself over the past couple years docker then Kubernetes and am now managing a small 3 node bare metal cluster on my own. But using rancher 2.0 has helped a bunch to ease me into it. Now I feel comfortable enough to start up a cluster on my own without it.

I strongly feel that Rancher 2.0 is just not ready for release yet. I run into so many weird edge cases using it, and don’t get me started on persistent storage... Longhorn is pretty easy to set up, but it seems to have some issues actually working (and slow).

I've had issues with longhorn as well, though the problem I was having (volume attachment race condition) is supposedly resolved by upgrading kubernetes.

I'm also trying out rook/ceph at the moment.

Re: Kubernetes Failure Stories

#197

Post by devthane »

qlk1123 wrote: post #139
devthane wrote: post #122

Earlier quoted context omitted.

This isn’t my experience at all. I as one person taught myself over the past couple years docker then Kubernetes and am now managing a small 3 node bare metal cluster on my own. But using rancher 2.0 has helped a bunch to ease me into it. Now I feel comfortable enough to start up a cluster on my own without it.

Just curious, is your cluster used in production or for any serious purpose?

Not for production yet. I'm in the process of moving our infrastructure from VMs that no one was really managing to Kubernetes. We're about to put together the production cluster and move over a couple of the applications that are fully ready and have been tested on the development cluster for a while.

Currently the cluster I'm talking about is being heavily used for QA testing purposes and Beta testing of applications. Kubernetes and a small operator I've written allows us to dynamically create QA servers for issues as developers finish their changes. Beta testing is easily deployed by a helm chart I wrote up.

Next we'll start up the production cluster and use it for internal applications and finally we'll be moving over our customer facing applications.

Re: Kubernetes Failure Stories

#198

Post by dgruesso »

bvm wrote: post #195
zerogvt wrote: post #183

Earlier quoted context omitted.

I think Istio ( https://istio.io ) is a nice effort to create both an abstraction on top of k8s and to package a set of commonly needed functionality out of the box. Unsure of its production status or overhead though. Also I'd only go with a managed k8s solution and I'm not sure I'd consider k8s for older or non-microservice/containerized architectures. In the later case though I don't think there's anything better o…

Lots of magic for me, I've broken a (dev) k8s cluster by installing Istio via Gitlab k8s integration. The overhead appeared to be non-negligible, but I noped-out-of-there pretty quick, so I don't have the data to back that up.

Hi bvm, GitLab PM here. Sorry to hear your dev cluster broke. Would like to offer any help we can provide and if possible learn more about the failure so we can take corrective action to avoid this in the future. Thanks.

Re: Kubernetes Failure Stories

#199

Post by manigandham »

ukoki wrote: post #188
manigandham wrote: post #125

Earlier quoted context omitted.

There is a very big gap between IaaS and PaaS. K8S is an abstraction on top of VMs so you can have a customizable PaaS that runs on YAML code. It has nothing to do with how complex your app is because K8S is about running it with less work in a declarative fashion. I'm currently in and have worked with dozens of startups that have saved lots of time by removing all the ops overhead with K8S because it runs the server…

"Every sufficiently large K8s deployment contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of a PaaS" K8s may well be the best foundation upon which to build a PaaS, but I think building a PaaS should be met with the same eye-rolls as writing your own crypto. Use Heroku or GAE or Elastic Beanstalk or IBM BlueMix or Azure PaaS or Cloud Foundry or Openshift.

It's just YAML files, there's nothing to build.

Re: Kubernetes Failure Stories

#200

Post by geggam »

dilyevsky wrote: post #151
aprdm wrote: post #147

Earlier quoted context omitted.

In places worked we usually had a vmware cluster, load balancer, NFS for shared data when necessary and DNS set up (e.g: through consul). This setup is very, very simple and scalable. There is very little to gain IMO on moving to Kubernetes. Consul, VSphere and load balancers have APIs and you can write tools to do everything that K8s does.

Scalable NFS, riiite.

worked at a company with 4 Petabytes on NFS ... FWIW
Post reply on HN