Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

srcco.de

Post reply on HN
Print view View original

11–20 of 242 posts

Re: Kubernetes Failure Stories

#11

Post by nicobn »

dehrmann wrote: post #10
tnolet wrote: post #3

I'd be interested in a related "microservices failure stories". Must be a big overlap with this.

I have two. One was caused by data inconsistency between services and regions. One is more hypothetical: the microservices had gotten to the point that no one knew how to start the system if all services are down, and it's possible that services have circular dependencies to the point that it would be incredibly hard to do a cold start.

I've actually seen your hypothetical in action, but the bug was even more subtle. Assume service A, B and C. A and C both need information from each other which is usually cached. Normally, you'd deploy one service at a time so the call chain would go A -> B -> C -> A or A -> C then A -> B -> C but in this particular instance, A and C's caches were cold, causing an explosion of service calls that took both services down.

Re: Kubernetes Failure Stories

#12

Post by lykr0n »

awinter-py wrote: post #8

Beyond strictly runtime failures, 2018 feels like the year that most of my friends tried kube but not everybody stayed on. The adoption failures are mostly networking issues specific to their cloud. Performance and box limits vary widely depending on cloud vendor and I still don't quite understand the performance penalty of the different overlay networks / adapters.

Network is a high performance system, and each layer you add adds latency.

Consider a traditional monolithic application. In comes your HTTP request in one end, a bunch of cross thread communication happens, and database queries come out the other end. With that, you have 2 points of network communication.

Now with a micro-service, you might have 4 or 5 applications that are needed to replace the above monolith. Throw in a service mesh on top of your cloud providers SDN, you've turned 2 points of network communication into 20 or more. The 5 micro-services talking to each other and the service meshes talking to each other. Add on top the additional processing overhead of maybe 1 to 2ms, you've just added at best 10ms round trip time to get to your databases and some more CPU. And to what benefit? TLS? You can do this in your application, or trust your private network is private. Tracing? You can do this with PID matching and watching the kernel's networking stack.

Re: Kubernetes Failure Stories

#13

Post by devereaux »

lykr0n wrote: post #12
awinter-py wrote: post #8

Beyond strictly runtime failures, 2018 feels like the year that most of my friends tried kube but not everybody stayed on. The adoption failures are mostly networking issues specific to their cloud. Performance and box limits vary widely depending on cloud vendor and I still don't quite understand the performance penalty of the different overlay networks / adapters.

Network is a high performance system, and each layer you add adds latency. Consider a traditional monolithic application. In comes your HTTP request in one end, a bunch of cross thread communication happens, and database queries come out the other end. With that, you have 2 points of network communication. Now with a micro-service, you might have 4 or 5 applications that are needed to replace the above monolith. Thro…

So true. For some low latency applications, anything above the bare minimal virtualization is not acceptable.

For what I do, in theory, many things should not impact results. In practice, anything that upon measurement impact results is stripped away. Think A/B testing but for every single component - including the major version of say the python interpreter.

That's how you end up running many things baremetal.

I'll say the future is not serverless but cloudless

Re: Kubernetes Failure Stories

#14

Post by hjacobs »

devereaux wrote: post #13
lykr0n wrote: post #12

Earlier quoted context omitted.

Network is a high performance system, and each layer you add adds latency. Consider a traditional monolithic application. In comes your HTTP request in one end, a bunch of cross thread communication happens, and database queries come out the other end. With that, you have 2 points of network communication. Now with a micro-service, you might have 4 or 5 applications that are needed to replace the above monolith. Thro…

So true. For some low latency applications, anything above the bare minimal virtualization is not acceptable. For what I do, in theory, many things should not impact results. In practice, anything that upon measurement impact results is stripped away. Think A/B testing but for every single component - including the major version of say the python interpreter. That's how you end up running many things baremetal. I'll…

I would argue that the longtail of applications does not really care about the impact of overlay networks. For us, the biggest impact on low-latency applications (where 1ms makes a difference) on Kubernetes was disabling CPU throttling in all clusters (you can also remove container limits). Background: a Kernel CFS quota bug leads to throttling even if quota is not yet reached, see https://www.youtube.com/watch?v=eBChCFD9hfs&feature=youtu.be...

Re: Kubernetes Failure Stories

#16

Post by lykr0n »

hjacobs wrote: post #14
devereaux wrote: post #13

Earlier quoted context omitted.

So true. For some low latency applications, anything above the bare minimal virtualization is not acceptable. For what I do, in theory, many things should not impact results. In practice, anything that upon measurement impact results is stripped away. Think A/B testing but for every single component - including the major version of say the python interpreter. That's how you end up running many things baremetal. I'll…

I would argue that the longtail of applications does not really care about the impact of overlay networks. For us, the biggest impact on low-latency applications (where 1ms makes a difference) on Kubernetes was disabling CPU throttling in all clusters (you can also remove container limits). Background: a Kernel CFS quota bug leads to throttling even if quota is not yet reached, see https://www.youtube.com/watch?v=eBC…

Overlay networks compound unstable networks. If your internal network latency spikes to 2ms from 0.5ms, normally that's not a huge issue but if you have micro-services that need to talk to each other- using my example, a 1.5ms round trip time would cause an additional 8ms of latency.

Sure, that's a low number but if you already have 150ms of processing, adding another 10ms might cause issues.

Also, if you're disabling the resource controls on kubernetes- you're kinda defeating the whole point.

Re: Kubernetes Failure Stories

#17

Post by hjacobs »

gspetr wrote: post #15
hjacobs wrote: post #2

Christian already followed the example and created a similar list for Serverless: https://github.com/cristim/serverless-failure-stories

Is there also a list for Docker failure stories?

IMHO this would be less interesting, some people already run other container runtimes such as containerd with Kubernetes (e.g. Datadog: https://www.youtube.com/watch?v=2dsCwp_j0yQ) --- so Docker might stay as some user interface for local development, but I would not know what "Docker failures" would be in the future.

Re: Kubernetes Failure Stories

#18

Post by lucidone »

coredog64 wrote: post #5
tnolet wrote: post #3

I'd be interested in a related "microservices failure stories". Must be a big overlap with this.

Microservices failure stories? “All of them. The End”

I'm consulting on a micro services back end right now with mostly prior experience with monoliths. What is the selling point that drives companies down this direction? It's insane, and my client keeps trying to hire new developers and bring on more consultants to build this thing, but the amount of knowledge required is more than any one person can handle. I have similar issues with their choice of db (nosql) and its inflexibility.

Re: Kubernetes Failure Stories

#20

Post by hjacobs »

awinter-py wrote: post #8

Beyond strictly runtime failures, 2018 feels like the year that most of my friends tried kube but not everybody stayed on. The adoption failures are mostly networking issues specific to their cloud. Performance and box limits vary widely depending on cloud vendor and I still don't quite understand the performance penalty of the different overlay networks / adapters.

> adoption failures are mostly networking issues specific to their cloud

Do you have any pointers/write-ups with more information or plans in this direction? I would be interested to learn more.

Post reply on HN