Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Common mistakes using Kubernetes

blog.pipetail.io

Post reply on HN
Print view View original

71–80 of 149 posts

Re: Common mistakes using Kubernetes

#71

Post by xrd »

fnord123 wrote: post #22
xrd wrote: post #13

I wish there was a way to upvote something 10x once a month here. This would be the post I use that on. When I was writing my book my editor asked me to remove any writing about mistakes and changes I made in the project for each chapter. I had a bug that appeared and I wanted to write about how I determined that and fixed it. They said the reader wants to see an expert talking, as if experts never make mistakes or n…

>They said the reader wants to see an expert talking, as if experts never make mistakes or need to shift from one tact to another. Your editor was very fucking wrong.

Obviously, I agree with you!

I also think this is the way the majority of tech books are written.

Can you think of another where the author goes from mistake to mistake and then finally gets it right?

I believe there is a space in tech writing for this kind of writing, but it is not something traditional book publishers believe.

This was an O'Reilly book by the way, with really good editors and a really good editoral process.

That editor was right most of the time, IMHO.

Re: Common mistakes using Kubernetes

#72

Post by sokoloff »

gridlockd wrote: post #54
fnord123 wrote: post #22

Earlier quoted context omitted.

>They said the reader wants to see an expert talking, as if experts never make mistakes or need to shift from one tact to another. Your editor was very fucking wrong.

> Your editor was very fucking wrong. The editor is completely right in what they were saying. You just want them to be wrong, because you'd prefer to live in the fantasy world where they are wrong. Let's say you go to get a surgery. You don't want the doctor to tell you about all the times they fucked up and what the awful consequences were. It doesn't matter that they're probably a better surgeon now, having learne…

I sure do want that surgeon to have been presented/instructed on the common ways that surgeons before them have made mistakes and how to avoid or overcome them.

That they won’t tell me (the patient) is quite a different question from whether they got the material from someone more experienced in their primary or continuing medical education.

Re: Common mistakes using Kubernetes

#73

Post by justinclift »

gridlockd wrote: post #54
fnord123 wrote: post #22

Earlier quoted context omitted.

>They said the reader wants to see an expert talking, as if experts never make mistakes or need to shift from one tact to another. Your editor was very fucking wrong.

> Your editor was very fucking wrong. The editor is completely right in what they were saying. You just want them to be wrong, because you'd prefer to live in the fantasy world where they are wrong. Let's say you go to get a surgery. You don't want the doctor to tell you about all the times they fucked up and what the awful consequences were. It doesn't matter that they're probably a better surgeon now, having learne…

> Let's say you go to get a surgery.

Shouldn't that be more like "Let's say you're learning to be a surgeon."?

For that situation, the person they're learning from discussing problems they hit and how they solved them does sound like it would be very useful.

Re: Common mistakes using Kubernetes

#74

Post by mlthoughts2018 »

jeffbee wrote: post #56
mlthoughts2018 wrote: post #27

I think there is more to the story for some of these points and it can be dangerous to just take this at face value of best practices. For example on the liveness / readiness probe item, the article says, > “ The other one is to tell if during a pod's life the pod becomes too hot handling too much traffic (or an expensive computation) so that we don't send her more work to do and let her cool down, then the readiness…

You of course should monitor the rate of liveness flapping for your services. The need to monitor it does not imply that it's a bad feature.

You can’t have it both ways. If you need to monitor it and take corrective action (which you do) then you shouldn’t rely on it.

This is an argument for making your liveness probe == readiness probe. It should just check pod availability in a minimal way, and if continuing to send the pod traffic based on this indicator turns out bad because of congestion, you want to see that causing errors and react, not let the scheduler take it out of service for new traffic.

You want liveness & readiness to check the same thing, and it should be a non-trivial check of service health that is also very low latency. And as long as that check is passing, keep sending traffic.

When the check fails, it should always be for a “hard down” reason that tells you the pod could not, regardless of traffic levels, accept traffic because it’s fundamentally internally down.

Re: Common mistakes using Kubernetes

#75

Post by fnord123 »

xrd wrote: post #71
fnord123 wrote: post #22

Earlier quoted context omitted.

>They said the reader wants to see an expert talking, as if experts never make mistakes or need to shift from one tact to another. Your editor was very fucking wrong.

Obviously, I agree with you! I also think this is the way the majority of tech books are written. Can you think of another where the author goes from mistake to mistake and then finally gets it right? I believe there is a space in tech writing for this kind of writing, but it is not something traditional book publishers believe. This was an O'Reilly book by the way, with really good editors and a really good editoral…

>Can you think of another where the author goes from mistake to mistake and then finally gets it right?

Not a book, but Raymond Hettinger often presents in this way and it's fantastic: https://www.youtube.com/watch?v=wf-BqAjZb8M

Re: Common mistakes using Kubernetes

#76

Post by jeffbee »

mlthoughts2018 wrote: post #74
jeffbee wrote: post #56

Earlier quoted context omitted.

You of course should monitor the rate of liveness flapping for your services. The need to monitor it does not imply that it's a bad feature.

You can’t have it both ways. If you need to monitor it and take corrective action (which you do) then you shouldn’t rely on it. This is an argument for making your liveness probe == readiness probe. It should just check pod availability in a minimal way, and if continuing to send the pod traffic based on this indicator turns out bad because of congestion, you want to see that causing errors and react, not let the sch…

I don't want the pager to go off just because of some slight non-liveness. That's a likely outcome of high utilization (usually viewed as a good thing, isomorphic with low cost). If you're running really hot and a few tasks are shedding load by playing dead intermittently, that's OK up to a point; if a large portion of pods are doing that at a high rate, that might be bad. You might not even alert on it, just throw it up on a dashboard as informative indicator for operators.

Re: Common mistakes using Kubernetes

#78

Post by sixhobbits »

This also needs a companion post called "common mistakes: using kubernetes".

I feel like it's a weekly occurrence now where I hear of a startup launching their mvp on kubernetes having spent 8 months too long on Dev as a result.

The other day in an interview someone bragged to me how he had convinced his team to spend 12 months moving to K8s. Upper management thought it was a waste of time but eventually agreed. I asked him if there were any measurable benefits and he said no.

I totally understands why Google needs it. Do you?

Re: Common mistakes using Kubernetes

#79

Post by yongjik »

tomnipotent wrote: post #12
haolez wrote: post #3

... requiredDuringSchedulingIgnoredDuringExecution: ... This instantly remembered me of this: https://thedailywtf.com/articles/the-longest-method Kubernetes sometimes shows its Java roots.

To be fair, I don't see how this could be shorter in any other language without losing readability.

You don't have to. Kubernetes should just say "required" here, and the documentation should say "Warning: this is only checked during scheduling."

printf isn't named printIntoBufferWhichMayNotFlushUntilLinefeed, and people are fine with it.

Re: Common mistakes using Kubernetes

#80

Post by yongjik »

sixhobbits wrote: post #78

This also needs a companion post called "common mistakes: using kubernetes". I feel like it's a weekly occurrence now where I hear of a startup launching their mvp on kubernetes having spent 8 months too long on Dev as a result. The other day in an interview someone bragged to me how he had convinced his team to spend 12 months moving to K8s. Upper management thought it was a waste of time but eventually agreed. I as…

Even Google doesn't need it that much: back when I was there, each Borg cluster had something like 10,000+ cores. Large enough to run a typical SV startup wholescale. The ratio of "cluster management work" vs. "actual work being done on it" was not that high.

These days, some people are like "Dude, if you don't have one cluster per AWS availability zone per each environment, you're doing it wrong." Why, just why.

Post reply on HN