Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go, Containers, and the Linux Scheduler

riverphillips.dev

Post reply on HN
Print view View original

131–140 of 143 posts

Re: Go, Containers, and the Linux Scheduler

#131

Post by otterley »

ithkuil wrote: post #108
eloisant wrote: post #103

Earlier quoted context omitted.

That's why you use monitor and alerting, so you notice degraded performances before the pods is crawling to a halt. You need to do it anyway because a service might progressively need more resources as it's getting more traffic, even if you're not adding any other pod.

Sure you need monitoring and alerting and sure there are other reasons why you need to update your requests. But having _neighbours_ affecting the behaviour of your workload is precisely what creates the kind of fatigue that then results in people claiming that it's hard to run k8s workloads. K8s is highly dynamical, pods can get scheduled on a node by chance sometimes and on some clusters; pagers will ring, incident…

> splitting up a large machine into smaller parts that can use the unused capacity of other parts in order to reduce waste is an advanced feature

That's an interesting take. For those of us who once used microcomputers back in the 1980s like Commodore 64s, Apple IIs, and IBM PCs running MS-DOS, sure. But time-sharing systems have been around since the 1960s and Unix dates back to the 1970s, and multi-user logins and resource controls like ulimits have been part of the picture for a very long time.

We've had plenty of time to get used to multitasking and resource contention and management. Is it complicated? It can be. But if you consider that containers are basically a program lifecycle/deployment and resource control abstraction on an OS that's had these features (maybe not cgroups and namespaces, but similar ones) since its birth, it's not really all that advanced.

Re: Go, Containers, and the Linux Scheduler

#132

Post by kevincox »

This is talking about containers but it seems that the problem is just whenever Go has access to less CPU time than it expects. Wouldn't the same thing happen when running Go on a system with another process that is using CPU? Or even just two Go programs at the same time?

Re: Go, Containers, and the Linux Scheduler

#133

Post by _cenw »

otterley wrote: post #90

This sort of tuning isn't necessary if you use CPU reservations instead of limits, as you should: https://home.robusta.dev/blog/stop-using-cpu-limits CPU reservations are limits, just implicit ones and declared as guarantees. So let the Go runtime use all the CPUs available, and let the Linux scheduler throttle according to your declared reservations if the CPU is contended for.

The Kubernetes community has this discussion every other week. This article isn't wrong per-se (and it's mostly content marketing, they have articles that are just plain wrong too, like https://home.robusta.dev/blog/containers-dont-use-chroot - i caused the "updated" tag on that one), but it's sweeping in its assertions, ignoring many good reason to set limits.

Workloads that use up all the bursting with little benefit, or prioritizing burst volume for an HTTP server instead of a cronjob that'll finish in designated time anyway. We even had a case where developers weren't updating their requests while their app grew in requirements, and then had an incident on their hand when spare CPU time suddenly was sparse.

Re: Go, Containers, and the Linux Scheduler

#134

Post by ithkuil »

otterley wrote: post #131
ithkuil wrote: post #108

Earlier quoted context omitted.

Sure you need monitoring and alerting and sure there are other reasons why you need to update your requests. But having _neighbours_ affecting the behaviour of your workload is precisely what creates the kind of fatigue that then results in people claiming that it's hard to run k8s workloads. K8s is highly dynamical, pods can get scheduled on a node by chance sometimes and on some clusters; pagers will ring, incident…

> splitting up a large machine into smaller parts that can use the unused capacity of other parts in order to reduce waste is an advanced feature That's an interesting take. For those of us who once used microcomputers back in the 1980s like Commodore 64s, Apple IIs, and IBM PCs running MS-DOS, sure. But time-sharing systems have been around since the 1960s and Unix dates back to the 1970s, and multi-user logins and…

The same arguments hold on old time-sharing systems. Setting up you limits in such a way that you can use excess capacity when available is very well suited (now and back then) for batch workloads. Scheduler priorities are a way to mix batch and interactive workloads.

When you're not latency sensitive but aiming at optimizing throughput you can achieve a pretty good utilization of the underlying resource. This is what resourceBatch workload still exists and it's important but the proportion of batch vs latency sensitive workload shifted significantly from the 70s to internet age. Request handlers that power that sit in the critical path for the user experience of a web or mobile app are not only latency sensitive but also _tail_ latency sensitive.

Tail latency often poses significant problems because stragglers can often determine the total duration of the interaction as perceived by the user, even if only one straggler request suffers from slowdowns.

While there are tricks to deal with tail latency while also improving utilization (i.e. reducing waste) they are hard to implement in practice and/or don't generalize well across workloads.

One thing always works and is your best option as a starting point: stop worrying about idle CPU in an _average_ usage metric. The behaviour during CPU usage spikes (even spikes that are shorter than your scraping period of your favourite monitoring tool!) determine latency envelope of your service. Measure how much CPU you need during those and plan accordingly.

Ideally it should be possible to some low priority (batch) process use that idle CPU, but AFAIK that's not currently possible with k8s.

EDIT: forgot to mention that all this discussion matters only inasmuch you have strict latency targets. If you're ok to occasionally have very slow requests and your customers are ok with that and your bosses don't freak out because they spoke to that one customer that is very angry that their requests time out a few times every month ... I'm very happy for you! Not everybody has these requirements and you can go pretty far with a production system consisting of a couple of VM on AWS without k8s, containers, cgroups or whatever in the picture. I know people who do and it works just fine for them. But in order to understand what the fuss about all of this we need to frame this discussion as pertaining to batch vs controlled-latency. Otherwise it's hard to explain why there are so many otherwise intelligent people making choices that appear to be a bit silly, isn't it? Sure, there are people who are "just wrong" on the internet :-) but more often than not if somebody ends up with a different solution is because they have different goals and tradeoffs.

Re: Go, Containers, and the Linux Scheduler

#135

Post by eightnoteight »

otterley wrote: post #90

This sort of tuning isn't necessary if you use CPU reservations instead of limits, as you should: https://home.robusta.dev/blog/stop-using-cpu-limits CPU reservations are limits, just implicit ones and declared as guarantees. So let the Go runtime use all the CPUs available, and let the Linux scheduler throttle according to your declared reservations if the CPU is contended for.

Reservations are not limits, they are minimum guaranteed CPU usage constraints. while theoretically, those are minimum guaranteed resources, if there are other busy containers running on the host, you will still see your tail latencies and average latency increasing abnormally

imagine you got 4 core ec2 instance, the latencies you see at 50% cpu utilisation and 90% cpu utilisation are quite different. with reservations similar thing happens i.e even though each container is guaranteed to their reservations, the relative CPU utilisation still increased very high when there are other busy processes on the same host

Re: Go, Containers, and the Linux Scheduler

#136

Post by eightnoteight »

eloisant wrote: post #103
ithkuil wrote: post #101

Earlier quoted context omitted.

I don't set limits because I'm afraid of how a pod is going to affect other pods. I set limits because I don't want to get used to being able to tap on the excess CPU available because that's not guaranteed to be available. As the node fills up with more and more other pods, it's possible that a pod that was running just fine a moment ago is crawling to a halt. Limits allow me to simulate the same behavior and plan f…

That's why you use monitor and alerting, so you notice degraded performances before the pods is crawling to a halt. You need to do it anyway because a service might progressively need more resources as it's getting more traffic, even if you're not adding any other pod.

imagine amazon's order history microservice causes outage for ordering creation microservice. regardless of the monitoring and alerting, you never want a non critical system to cause outage on the most critical system you have

Re: Go, Containers, and the Linux Scheduler

#137

Post by eightnoteight »

dilyevsky wrote: post #55
the8472 wrote: post #41

Earlier quoted context omitted.

With shares you're going to experience worse latency if all the containers on the system size their thread pool to the maximum that's available during idle periods and then constantly context-switch due to oversubscription under load. With quotas you can do fixed resource allocation and the runtimes (not Go apparently) can fit themselves into that and not try to service more requests than they can currently execute g…

And how is that different from worse latency due to cpu throttling from your users’ perspective?

this is a predictable worse latency due to CPU throttling, i.e nothing is suddenly introduced in the system. but the other case is worse i.e a non critical microservice can cause outage on your critical microservice.

imagine some non critical system like blog-service suddenly causing 2-3% new order creation failures

Re: Go, Containers, and the Linux Scheduler

#138

Post by demi56 »

crabbone wrote: post #124
sofixa wrote: post #102

Earlier quoted context omitted.

> VMware fumbled the bag Oh they did, they're a modern day IBM. > Containers are a crappy abstraction They're one of the best abstractions we have (so far) because they contain only the application and what it needs.

> they contain only the application and what it needs. Delusion level: over 9900. I'm yet to find a container that contains only the application and what it needs. Most of the time I find that they contain at least libc and libpthreads (which are already present on the host, so not needed). More often I find metric tonnes of garbage that was not necessary by any metric, but was just too hard to remove, so was allowed…

Blame that to glibc not the container

Re: Go, Containers, and the Linux Scheduler

#139

Post by fhuici »

marcus_holmes wrote: post #60

I still don't get the benefit of running Go binaries in containers. Totally get it for Rails, Python, etc, where the program needs a consistent set of supporting libraries. But Go doesn't need that. Especially now we can embed whole file systems into the actual binary. I've been looking at going in the other direction and using a Go binary with a unikernel; the machine just runs the binary and nothing else. I haven't…

Hi, one of the maintainers of the Unikraft LF OSS project here (www.unikraf.org), clearly in agreement :) . We regularly run Go workloads, and even use Dockerfiles to build Go and other projects that we then turn into minimal unikernels. We also have a closed beta/free (unikernel) cloud platform if people want to try, sign up is at kraft.cloud .

Re: Go, Containers, and the Linux Scheduler

#140

Post by fhuici »

marcus_holmes wrote: post #74
throwaway892238 wrote: post #70

Earlier quoted context omitted.

Here's what a container gives you: - Isolation of networking - Isolation of process namespace - Isolation of filesystem namespace - CPU and Memory limit enforcement - A near-universal format for packaging, distributing, and running applications, with metadata, and support for multiple architectures - A simple method for declaring a multi-step build process - Other stuff I don't remember You're certainly welcome to go…

A VM provides the first 4 anyway - if you're deploying to a cloud instance then having these in the container is redundant. If you're deploying to bare metal then it's possibly useful, but only if you're deploying multiple containers to the same machine. Go doesn't need a format for packaging - it's one file. It's becoming common practice to embed everything else into the binary. (side note: I haven't done this with…

At Unikraft (OSS unikernel project) we do a bit of both: if wanted, people can specify via a Dockerfile what they want/need in their filesystem, and then we have a tool that compiles (if needed) and packs the files into a (OCI formatted) unikernel (which we then deploy via kraft.cloud ).
Post reply on HN