Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

srcco.de

Post reply on HN
Print view View original

161–170 of 242 posts

Re: Kubernetes Failure Stories

#161

Post by apaz037 »

karakanb wrote: post #103
tyingq wrote: post #43

Earlier quoted context omitted.

Digital Ocean's K8S offering is out of beta now: https://www.digitalocean.com/products/kubernetes/

Migrated my very small cluster from GKE to DigitalOcean's K8s a few weeks ago. I was using 3 nodes on GKE with 1 core & 3.75GB RAM per node, and the cost was around 100 $ per month including load balancer for the cheapest region, `us-central1-a`. Now, on DigitalOcean, I have 3 nodes with 1 core & 2GB RAM per node. The cost is exactly 40$ including load balancer. I am a pretty basic user, I have started using k8s on t…

The problem with that is that I can almost guarantee that it would still be cheaper and easier to manage if you just leveraged whatever cloud provider's managed service was there to run your stuff.

Re: Kubernetes Failure Stories

#162

Post by dilyevsky »

aprdm wrote: post #154
dilyevsky wrote: post #151

Earlier quoted context omitted.

Scalable NFS, riiite.

If you have some time to read "how Google works" you would be surprised by how long the company ran on NFS. I assume there are lots of workloads running on Borg to this day on top of NFS. If that isn't enough for you you should have a look in the client list of Isilon and see which kind of work they do, in case you ever attend a SIGGRAPH most of what you see is built on top of NFS, so, essentially, all of the compute…

Sorry I don’t have to read it because i was borg sre for 6 years and i know how (the server part of) it works. You assume wrong.

I know there are a lot of companies that try to put some lipstick on nfs pig and call it reliable/scalable/etc. so long their clients don’t actually try to run it at scale or don’t complain too publicly when they try and can’t, they are able to get away with it.

Re: Kubernetes Failure Stories

#163

Post by akhilcacharya »

bpicolo wrote: post #101
jacques_chester wrote: post #79

Earlier quoted context omitted.

Kubernetes has always had an identity crisis. Who is aimed at, app developers or platform operators? Clear, obvious contracts between the two roles are valuable, even if you decide to combine them. I'm moderately hopeful that Knative will help in that regard, as it is more conclusively oriented towards the developer. But I am wary that since it leaves the implementation details completely visible, it may not achieve…

> app developers or platform operators Definitely not the former. The YAML-based configuration is not a pleasant app deployment experience. Companies end up needing to do some sort of auto-generation for it to make it sane for app devs. App developers want experiences similar to heroku. They want to git push and have applications safely roll out without downtime or configuration.

I’m a little behind on Cloud Native adoption so I gotta ask - what’s preventing a “git push to heroku” from being the norm here? Are we using the wrong abstractions? Or are the abstractions still too low?

...why is Heroku/buildpack not running away with it?

Re: Kubernetes Failure Stories

#164

Post by aprdm »

dilyevsky wrote: post #162
aprdm wrote: post #154

Earlier quoted context omitted.

If you have some time to read "how Google works" you would be surprised by how long the company ran on NFS. I assume there are lots of workloads running on Borg to this day on top of NFS. If that isn't enough for you you should have a look in the client list of Isilon and see which kind of work they do, in case you ever attend a SIGGRAPH most of what you see is built on top of NFS, so, essentially, all of the compute…

Sorry I don’t have to read it because i was borg sre for 6 years and i know how (the server part of) it works. You assume wrong. I know there are a lot of companies that try to put some lipstick on nfs pig and call it reliable/scalable/etc. so long their clients don’t actually try to run it at scale or don’t complain too publicly when they try and can’t, they are able to get away with it.

Your concept of what is scale looks very different than mine, in my experience NFS does a very good job for in-datacenter workloads. CG rendering, oil/gas and others usually take this approach for HPC as far as I've seen. I consider this "scale". Close to 100k procs sharing the nfs is the biggest cluster I've worked at.

Of course that over longer networks it isn't suitable as the roundtrips have too much latency, other than that, is your experience much different regarding nfs?

Re: Kubernetes Failure Stories

#165

Post by cookiecaper »

dlor wrote: post #136
cookiecaper wrote: post #120

Earlier quoted context omitted.

k8s is meant to be hard to use. You're supposed to rent space on a k8s cluster from Google. Google has been pumping millions into marketing k8s as a mechanism to improve GCP adoption and establish a foothold in the cloud provider space.

I'm not exactly sure what point you're trying to make here. k8s is not meant to be a paas, but no one is trying to make k8s harder to use. I work at Google on a large team of engineers dedicated to making it as easy as possible to use.

[Disclaimer: this is pure conjecture and represents my opinion only; I'm a Google outsider.]

For a while, "AWS" and "the cloud" were practical synonyms; it was very rare that anyone meant not-AWS. In my opinion, Kubernetes is a major piece of Google's strategy to turn that tide and improve their marketshare.

Does throwing "a large team of engineers" at a problem typically result in something that's "as easy as possible to use"? "Design-by-committee" is not a term of endearment.

A sibling comment at https://news.ycombinator.com/item?id=18958077 notes that "there are a ton of nicer UIs for Kubernetes", but that they're sold separately as proprietary PaaS platforms. Even if we pretend like GCP/GKE isn't one of them, Kubernetes-The-Platform will be impacted by the interests of its primary vendors and advocates, whether or not certain teams at Google are keen to admit that.

Re: Kubernetes Failure Stories

#166

Post by verisimilidude »

atgreen wrote: post #140
verisimilidude wrote: post #126

Earlier quoted context omitted.

Simplified Kubernetes is a thing that exists. OpenShift (and the open source version, OKD) jumps out as the immediate example. There are other non-k8s tools that cover some of the same territory, like Docker Swarm or Cloud Foundry. There's still a learning curve, but it's much more humane than Kubernetes.

I think you meant to write "(and the upstream community version, OKD)", because OpenShift is also fully open source.

Yes, many thanks for the correction.

Re: Kubernetes Failure Stories

#167

Post by jordanbeiber »

lmm wrote: post #123
jordanbeiber wrote: post #117

Earlier quoted context omitted.

We view this a bit different, and that’s fine (not the automation part, here we agree). Automation in this context is for me more than just the deploy bit, it’s also about testing and service management which includes for example service relationships and discovery. If you could do local dev on that app as a monolith, it can most likely be done broken up in smaller services as well. There are no silver bullets to be…

> If you could do local dev on that app as a monolith, it can most likely be done broken up in smaller services as well. It's possible, but the overhead is a lot higher, and that weighs down everything you do. Your edit-test cycle gets longer, development gets slower. > There are no silver bullets to be had anywhere, right? Just use whatever processes and tools that work, until they don’t I guess. Nothing is perfect…

Yes, there is bound to be overhead as you describe. For some it might be worth the effort, but no doubt effort is involved.

Regarding monolith -> ms — I’m starting to think this is the way to do it — once you have the flow of data already estblished and the (working) system(s) somewhat defined it becomes easier to decouple bits and pieces.

Re: Kubernetes Failure Stories

#168

Post by ithkuil »

justicezyx wrote: post #138
malkia wrote: post #132

Earlier quoted context omitted.

It took me a while to get comfortable in Borg (and in general that your binary can take hundredths of verbosely written command-line arguments (coming from gamedev, I was in a bit of shock state for a while)... But then got used to it - still I felt I could never fully internalize the evaluation rules - but the other tooling (diffing) really helped in that respect. One thing I've really appreciated, was how one could…

You seem confuse Borg and borgcfg. The evaluation rules are merely a borgcfg artifact. Disclaimer: I maintain borgcfg.

I found https://jsonnet.org/ to fix several issues with the Borg configuration language.

https://github.com/ksonnet/kubecfg is an attempt to reboot the borgcfg experience with k8s + jsonnet

Re: Kubernetes Failure Stories

#169

Post by kortilla »

m0zg wrote: post #156
aprdm wrote: post #154

Earlier quoted context omitted.

If you have some time to read "how Google works" you would be surprised by how long the company ran on NFS. I assume there are lots of workloads running on Borg to this day on top of NFS. If that isn't enough for you you should have a look in the client list of Isilon and see which kind of work they do, in case you ever attend a SIGGRAPH most of what you see is built on top of NFS, so, essentially, all of the compute…

82gb/s (assuming you mean gigabit) is _per-node_ throughput at Google (or FB, or I assume Amazon/Microsoft -- they all use 100GbE networks now). 300K IOPS is probably per-node, too, at this point. :-)

Having a 100gbps nic in a node isn’t the same thing as doing storage at that speed in an HA cluster.

Also, don’t confuse 100 gbe networks where spine links are 100 but the node links are only bonded 10s (much more common at $fang).

Re: Kubernetes Failure Stories

#170

Post by jordanbeiber »

jacques_chester wrote: post #129
jordanbeiber wrote: post #117

Earlier quoted context omitted.

We view this a bit different, and that’s fine (not the automation part, here we agree). Automation in this context is for me more than just the deploy bit, it’s also about testing and service management which includes for example service relationships and discovery. If you could do local dev on that app as a monolith, it can most likely be done broken up in smaller services as well. There are no silver bullets to be…

In think your interlocutor's point is that type checking by a compiler is a lot simpler, more reliable and more performant than any networked service discovery scheme so far conceived. I think microservice architectures can genuinely decouple teams to iterate faster and consolidate efforts. But it's not a free lunch.

I totally agree.

What I was trying to say is that _if_ you go down the many services route, a lot of automation and integrations will be needed, and perhaps in completely new places.

No free lunches ever... just a lot of hard work. :)

Post reply on HN