Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go, Containers, and the Linux Scheduler

riverphillips.dev

Post reply on HN
Print view View original

81–90 of 143 posts

Re: Go, Containers, and the Linux Scheduler

#81

Post by n3t »

jeffbee wrote: post #80
n3t wrote: post #78

Earlier quoted context omitted.

If by "CPU mask" you refer to the `sched_getaffinity` syscall, I can't reproduce this behavior. What I tried: I created a "Burstable" Pod and run `nproc` [0] on it. It returned N CPUs (N > 1). Then I created a "Guaranteed QoS" Pod with both requests and limit set to 1 CPU. `nproc` returned N CPUs on it. I went back to the "Burstable" Pod. It returned N. I created a fresh "Burstable" Pod and run `nproc` on it, got N a…

I don't know what nproc does. Consider `taskset`

I re-did the experiment again with `taskset` and got the same results, i.e. the mask is independent of creation of the "Guaranteed QoS" Pod.

FWIW, `taskset` uses the same syscall as `nproc` (according to `strace`).

Re: Go, Containers, and the Linux Scheduler

#82

Post by jeffbee »

n3t wrote: post #81
jeffbee wrote: post #80

Earlier quoted context omitted.

I don't know what nproc does. Consider `taskset`

I re-did the experiment again with `taskset` and got the same results, i.e. the mask is independent of creation of the "Guaranteed QoS" Pod. FWIW, `taskset` uses the same syscall as `nproc` (according to `strace`).

Perhaps it is an artifact of your and my various container runtimes. For me, in a guaranteed qos pod, taskset shows just 1 visible CPU for a Guaranteed QoS pod with limit=request=1.

  # taskset -c -p 1
  pid 1's current affinity list: 1

  # nproc
  1
I honestly do not see how it can work otherwise.

Re: Go, Containers, and the Linux Scheduler

#83

Post by dilyevsky »

marcus_holmes wrote: post #74
throwaway892238 wrote: post #70

Earlier quoted context omitted.

Here's what a container gives you: - Isolation of networking - Isolation of process namespace - Isolation of filesystem namespace - CPU and Memory limit enforcement - A near-universal format for packaging, distributing, and running applications, with metadata, and support for multiple architectures - A simple method for declaring a multi-step build process - Other stuff I don't remember You're certainly welcome to go…

A VM provides the first 4 anyway - if you're deploying to a cloud instance then having these in the container is redundant. If you're deploying to bare metal then it's possibly useful, but only if you're deploying multiple containers to the same machine. Go doesn't need a format for packaging - it's one file. It's becoming common practice to embed everything else into the binary. (side note: I haven't done this with…

Are you suggesting to setup a separate VM for each process that may only require like 0.25 cpu? Another thing you can’t do with VMs is oversubscribe (at least not with cloud ones)

Re: Go, Containers, and the Linux Scheduler

#84

Post by n3t »

jeffbee wrote: post #82
n3t wrote: post #81

Earlier quoted context omitted.

I re-did the experiment again with `taskset` and got the same results, i.e. the mask is independent of creation of the "Guaranteed QoS" Pod. FWIW, `taskset` uses the same syscall as `nproc` (according to `strace`).

Perhaps it is an artifact of your and my various container runtimes. For me, in a guaranteed qos pod, taskset shows just 1 visible CPU for a Guaranteed QoS pod with limit=request=1. # taskset -c -p 1 pid 1's current affinity list: 1 # nproc 1 I honestly do not see how it can work otherwise.

After reading https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana..., I think we have different policies set for the CPU Manager.

In my case it's `"cpuManagerPolicy": "none"` and I suppose you're using `"static"` policy.

Well, TIL. Thanks!

Re: Go, Containers, and the Linux Scheduler

#85

Post by wonrax »

mholt wrote: post #66
marcus_holmes wrote: post #60

I still don't get the benefit of running Go binaries in containers. Totally get it for Rails, Python, etc, where the program needs a consistent set of supporting libraries. But Go doesn't need that. Especially now we can embed whole file systems into the actual binary. I've been looking at going in the other direction and using a Go binary with a unikernel; the machine just runs the binary and nothing else. I haven't…

As the author of Caddy, I have often wondered why people run it in containers. The feedback I keep hearing is basically workflow/ecosystem lock-in. Everything else uses containers, so the stuff that doesn't need containers needs them now, too.

For my personal site I run Caddy in a Docker container along with other containers using a compose file. By doing it this way, getting things up when moving to a new instance is as simple as running `docker compose up`. Also making changes to the config or upgrading Caddy version on deployments is the same as a other services since they're all containers. So it's easy to add CI/CD and have it re-deploy Caddy whenever the config changes and there's no need for extra GitHub Actions .yaml's. Setup as code like this also documents all the dependencies and I think it might be helpful in the future.

Having said that, for serious business, this setup doesn't make sense. It possibly takes more work to operate as container when the gateway runs on a dedicated instance.

Re: Go, Containers, and the Linux Scheduler

#86

Post by richdougherty »

yjftsjthsd-h wrote: post #37
bruh2 wrote: post #29

As someone not that familiar with Docker or Go, is this behavior intentional? Could the Go team make it aware of the CGroups limit? Do other runtimes behave similarly?

I'm fairly certain that that .net had to deal with it and Java had or still has a problem, I forget which. (Or did you mean runtimes like containerd?)

Supported in Java 10 (and backported to Java 8) since 2018. Not sure about .NET.

- "The JVM has been modified to be aware that it is running in a Docker container and will extract container specific configuration information instead of querying the operating system. The information being extracted is the number of CPUs and total memory that have been allocated to the container." https://www.oracle.com/java/technologies/javase/8u191-relnot...

- Here's a more detailed explanation and even a shared library that can be used to patch container unaware versions of Java. I wonder if the same could be done for Go?

"LD_PRELOAD=/path/to/libproccount.so java "

https://stackoverflow.com/a/64271429

https://gist.github.com/apangin/78d7e6f7402b1a5da0fa3abd9381...

-

There are more recent changes to Java container awareness as well:

https://developers.redhat.com/articles/2022/04/19/java-17-wh...

Re: Go, Containers, and the Linux Scheduler

#87

Post by dilyevsky »

jeffbee wrote: post #63
dekhn wrote: post #59

Earlier quoted context omitted.

It's complicated. I've worked on every kind of application in a container environment: ones that ran at ultra-low priority while declaring zero CPU request and infinite CPU limit . I ran one or a few of these on nearly every machine in Google production for over a year, and could deliver over 1M xeon cores worth of throughput for embarassingly parallel jobs. At other times, I ran jobs that asked for and used precisel…

The problem is your experience involves a hacked up Linux that was far more suitable for doing this than is the upstream. Upstream scheduler can't really deal with running a box hot with mixed batch and latency-sensitive workloads and intentionally abusive ones like yours ;-) That is partly why kubernetes doesn't even really try.

This. Some googlers forget there is a whole team of kernel devs in TI that are maintaining patched kernel (including patched CFS) specifically for Borg

Re: Go, Containers, and the Linux Scheduler

#88

Post by usr1106 »

ecnahc515 wrote: post #76
pbh101 wrote: post #61

Earlier quoted context omitted.

CFS quotas only become active under contention and even then are relative: if you’re the only thing running on the box and want all the cores but only set one cpu, you get all of them anyway. If you set cpus to 2 and another process sets to 1 and you both try to use all CPUs all out, you’ll get 66% and they’ll get 33%. This isn’t the same as cpusets, which work differently.

> CFS quotas only become active under contention That's not true at all. Take a look at `cpu.cfs_quota_us` in https://kernel.googlesource.com/pub/scm/linux/kernel/git/glo... It's a hard time limit. It doesn't care about contention at all. `cpu.shares` is relative, for choosing which process gets scheduled, and how often, but the CFS quota is a hard limit on runtime.

Yes, there are hard limits in the CFS. I have used them for thermal reasons in the past, such that the system remained mostly idle although some threads would have had more work to do.

Not at my work environment right now, don't remember the parameters I used.

Re: Go, Containers, and the Linux Scheduler

#89

Post by lmm »

dilyevsky wrote: post #83
marcus_holmes wrote: post #74

Earlier quoted context omitted.

A VM provides the first 4 anyway - if you're deploying to a cloud instance then having these in the container is redundant. If you're deploying to bare metal then it's possibly useful, but only if you're deploying multiple containers to the same machine. Go doesn't need a format for packaging - it's one file. It's becoming common practice to embed everything else into the binary. (side note: I haven't done this with…

Are you suggesting to setup a separate VM for each process that may only require like 0.25 cpu? Another thing you can’t do with VMs is oversubscribe (at least not with cloud ones)

You can't have both oversubscription and isolation, almost by definition. If you want isolation, VMs are great. If you want oversubscription, OSes are still better than container runtimes at managing competing processes on the same host.

Re: Go, Containers, and the Linux Scheduler

#90

Post by otterley »

This sort of tuning isn't necessary if you use CPU reservations instead of limits, as you should: https://home.robusta.dev/blog/stop-using-cpu-limits

CPU reservations are limits, just implicit ones and declared as guarantees.

So let the Go runtime use all the CPUs available, and let the Linux scheduler throttle according to your declared reservations if the CPU is contended for.

Post reply on HN