Listen to Grace Hopper. Count the feet (nanoseconds) data has to travel at speed of light so the Admiral respects the laws of physics.
Blending complex systems made my latency 10x higher
41–50 of 86 posts
Re: Blending complex systems made my latency 10x higher
#42And this is why I find DevOps work more interesting than programming. System integration is just endlessly challenging. I always enjoy reading a well-documented integration debugging session!
In my opinion DevOps shouldn't exist at all.
I think the best model is to have your "devops/sre guy" embedded right on the team.
I market myself in interviews as a software engineer (on the same team as the other SWEs) that just happens to focus on what actually getting this product into production and keeping it going looks like.
I got my start in this by working at a startup where we were responsible for everything just by the nature of the limited staff. I think that's how a lot of people in Devops get their start.
I didn't even know coding without a thought about deployment was a thing until I joined a huge company.
Re: Blending complex systems made my latency 10x higher
#43Earlier quoted context omitted.
In my opinion DevOps shouldn't exist at all.
Okay. So based on 20+ years of experience, I can say that most developers have no interest in automating deployment, configuration, monitoring, performance, logging, etc... Who should do this work?
Yeah, operations-focused engineers will continue to have a niche carved out for them because too many devs black-box infrastructure.
Companies can either choose to have their devs take on ops responsibilities or continue having dedicated ops jobs.
In either case, whether or not dedicated ops jobs exist, ops responsibilities always will. I'll be there to pick up the slack because designing and maintaining systems is an interesting job that has to be done that a lot of people can't/won't do and it pays accordingly.
Re: Blending complex systems made my latency 10x higher
#44Earlier quoted context omitted.
In my opinion DevOps shouldn't exist at all.
Okay. So based on 20+ years of experience, I can say that most developers have no interest in automating deployment, configuration, monitoring, performance, logging, etc... Who should do this work?
Interesting, I've observed many teams where the engineers seem to prefer the DevOps work to actually building features. To the point where I think some times we have too many engineers Opsing rather than Deving.
Re: Blending complex systems made my latency 10x higher
#45this is good info but the title is misleading. This could have easily been "my latency increased because my local time was off" or something like that.
This had nothing to do with kubernetes..it was a problem with their set up.
Re: Blending complex systems made my latency 10x higher
#46And this is why I find DevOps work more interesting than programming. System integration is just endlessly challenging. I always enjoy reading a well-documented integration debugging session!
In my opinion DevOps shouldn't exist at all.
So should developers be doing all their own ops work in a half-assed, ignorant way, or should we go back to a world of throw it over the wall systems where the ops team doesn't understand the code or have a working non-adversarial relationship with the developers?
Because I've lived with both, and to hell with both of them.
Re: Blending complex systems made my latency 10x higher
#47What grinds my gears is hard-coded magic timeout numbers. Somehow microservice people seem to think that these are good (e.g. for circuit breakers) without seeming to realize the unexpected consequences of composing like this. Your timeout is not my timeout. So firstly - don't do it. Time is an awful thing to build behaviour on, and secondly if you ignore that then make it a config parameter - then I've got a chance…
The timeouts this post is talking about are related to credential expiration and when to refresh, not request/connection timeouts like you'd see in microservices. In this case, not expiring credentials isn't a great option because you'd lose a useful security property: Reducing the time window when stolen credentials can be used.
For service behavior (e.g. request/connections), timeouts provide value for services and clients. For services, if you never time out then under failure conditions you either end up saturating your max concurrent request limits or growing your concurrent connections indefinitely until you hit a limit (connections, threads, RAM, CPU). Unless all of your clients are offline batch processes with no latency SLA there's a good chance that the work clogging up your service was abandoned by your clients long before it completes.
Timeouts also help clients decide if/when they should retry. Even if the service never times out, clients can't really tell if their request is just taking a long time or if something is wrong and the request will never succeed (e.g. network partitions, hardware failure). There's at least implicit latency SLAs on most things we do (1 second? how about an hour or week?). Given that there is a limit somewhere, it makes sense to use that limit to get benefits like resiliency in services.
>>>Your timeout is not my timeout.
Absolutely. Client deadlines are a great way to reduce wasted work across distributed systems. e.g. service has 60s timeout, but client has a 5s timeout for an online UI interaction. The client can tell the service to give up if it's not completed within the client's SLA.
Re: Blending complex systems made my latency 10x higher
#48Agreed about poor title, but: > DNS resolution is indeed a bit slower in our containers (the explanation is interesting, I will leave that for another post). I would like to see this expanded upon or hear if anyone else has suffered similar.
Exactly! On our OpenShift production cluster we ran into ndots problems with DNS and slow DNS resolution overall. This blog post was very helpful in understanding the issue and ways to fix it, https://pracucci.com/kubernetes-dns-resolution-ndots-options...
Yeah, Tim Hockin and I still regret not designing the DNS name search process in Kube better. If we had, we would have avoided the need for 5 and could have kept 90% of the usability win of “name.namespace.svc” and “name” being resolvable to services without having to go to 5. And now we can’t change it by default without breaking existing apps.
Forwards compatibility is painful.
Re: Blending complex systems made my latency 10x higher
#49Earlier quoted context omitted.
It wasn’t. It was KIAM and the Java SDK, neither of which are part of K8S.
KIAM is not part of the core Kubernetes system but it was a necessary component to avoid introducing a security regression as part of the switch. Again, my point was that rather than trying to do PR damage-control it would be better to work to improve things so this doesn't understand. Someone went to the trouble of posting a detailed examination of a real problem with a fix and some references to upstream improvemen…
It’s not PR. It’s being accurate that this has nothing to do with Kubernetes.
Re: Blending complex systems made my latency 10x higher
#50And this is why I find DevOps work more interesting than programming. System integration is just endlessly challenging. I always enjoy reading a well-documented integration debugging session!
In my opinion DevOps shouldn't exist at all.
Can you elaborate? You may as well say “software engineering should not exist at all”.