Live data from Hacker News

Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

bugs.launchpad.net

91–100 of 175 posts

Re: Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

#91
post #2

It's not just a bunch of people setting cm.swappieness to 0 is it? That behaviour changed and now prevents going into swap /ever/ where it used to mean go into swap as a last resort. Changing it to 1 fixes the issue and will act like the old behaviour or only swapping as a last result. Regardless, it's things like this why we never recommend people use Ubuntu - years of poorly packaged and tested software. If you wan…

Don't use EPEL in production, it is an unstable mess. If you need more packages than are in the core of CentOS, Debian is the best choice.

Re: Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

#92
post #3
post #2

It's not just a bunch of people setting cm.swappieness to 0 is it? That behaviour changed and now prevents going into swap /ever/ where it used to mean go into swap as a last resort. Changing it to 1 fixes the issue and will act like the old behaviour or only swapping as a last result. Regardless, it's things like this why we never recommend people use Ubuntu - years of poorly packaged and tested software. If you wan…

Can you please share your experience a bit more ? How much SELinux is really required ? What about grsecurity. My use case is custom build Django/Postgres app.

SELinux is an absolute must in todays world, simply put it prevents applications and namespace from doing thing outide of their area of interet, for example a web server should never be allowed to execute files from /tmp so it prevents that, another great example is when you have any form of multi-tennancy - for example you might have several docker or LXC containerised apps running on a host SELinux will / can ensure that even if of those apps is compromised and someone popped a shell in it, they couldn't go beyond anything running in that namespace, i.e. to the host itself or other apps running in containers on the same host.

It prevents a lot of sitations where an app might behave badly, for example perhps you're running a PHP app and a 0-day PHP flaw is discovered, there's a good chance that SELinux restrictions could prevent that exploit from being useful to the attacker by not allowing the app to do unusual things.

We have a rule - if it's not running SELinux in enforicng then it's not going into production.

We use to run GRSec as well, however with modern kernels a lot of the GRSec code base has been merged in so it's often not required.

Re: Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

#93
post #4
post #2

It's not just a bunch of people setting cm.swappieness to 0 is it? That behaviour changed and now prevents going into swap /ever/ where it used to mean go into swap as a last resort. Changing it to 1 fixes the issue and will act like the old behaviour or only swapping as a last result. Regardless, it's things like this why we never recommend people use Ubuntu - years of poorly packaged and tested software. If you wan…

Centos/rhel are on a pretty old kernel at this point though?

We run the latest mainline kernel on all our 200-300ish CentOS hosts after it's passed standard testing and is available in Elrepo: http://elrepo.org/tiki/kernel-ml

Back when we ran Debian everywhere I wrote a CI project that would build the latest mainline kernel, applying GRSec and adding some SELinux policies in as best I could (backported from CentOS/Fedora funnily enough) and build them into a Debian package for deployment across our hosts, see: https://github.com/sammcj/kernel-ci

Re: Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

#94
post #75

Earlier quoted context omitted.

For me swap is an error. I would rather a server crash then go into swap and run slow. If your servers are at 75% CPU and start swapping you are screwed. Time to start adding servers. In this day and age you are wasting a lot of money to be sitting around waiting for swap.

zswap makes swap useful on the server as it results in gradual slowdown when the system has used all the ram. And if the system runs at 75% of CPU, then there is room for such slowdown.

I hadn't heard of zswap [0]:

"Zswap is a lightweight compressed cache for swap pages. It takes pages that are in the process of being swapped out and attempts to compress them into a dynamically allocated RAM-based memory pool. [...] Zswap is a new feature as of v3.11 [...] zswap is a work in progress and should be considered experimental."

The return of RAM Doubler!

The naming is initially confusing if you're already familiar with ZFS, especially:

"Zswap makes use of zpool for the managing the compressed memory pool"

[0] https://www.kernel.org/doc/Documentation/vm/zswap.txt

Re: Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

#95
post #4

Earlier quoted context omitted.

Centos/rhel are on a pretty old kernel at this point though?

For Centos 7 its 3.10 according to good old http://distrowatch.com/table.php?distribution=centos

and it's whatever is the latest, stable, tested version of the kernel when using Elrepo's Kernel-ml repo: http://elrepo.org/tiki/kernel-ml

Re: Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

#96
post #2

It's not just a bunch of people setting cm.swappieness to 0 is it? That behaviour changed and now prevents going into swap /ever/ where it used to mean go into swap as a last resort. Changing it to 1 fixes the issue and will act like the old behaviour or only swapping as a last result. Regardless, it's things like this why we never recommend people use Ubuntu - years of poorly packaged and tested software. If you wan…

What the hell? What happened to "don't break userspace"?

Yeah I know right, Ubuntu breaks userspace all the time, Debian isn't quite as bad but it's much worse than it used to be thanks to so many missing packages, try building a cluster / HA system on it - you actually can't!

Re: Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

#97
post #15
post #2

It's not just a bunch of people setting cm.swappieness to 0 is it? That behaviour changed and now prevents going into swap /ever/ where it used to mean go into swap as a last resort. Changing it to 1 fixes the issue and will act like the old behaviour or only swapping as a last result. Regardless, it's things like this why we never recommend people use Ubuntu - years of poorly packaged and tested software. If you wan…

Crikey - they did what?! I recall a fascinating article / thread years ago about swappiness=0 not actually meaning what people thought it meant, and explaining why it was (generally) a bad idea. A low number, yes, but not 0. Doing some research now, and found a blog written by someone bitten by this change back in 2014-04 -- referring to the commit that introduced this in 3.5-rc1. Not clear why the change was made, t…

Yeah it wasn't well announced when the change was made, esentially if you find any linud machines running any kernel dating way back to 3.5 make sure you change vm.swappiness=0 to vm.swappiness=1

Re: Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

#98

Earlier quoted context omitted.

Swap makes a lot of sense for a consumer device, where I'm opening a big spreadsheet, then putting that in background to edit this video. It makes less sense for server workloads, where you usually don't have huge chunks of inactive memory. I always run with no swap in AWS, especially with the newer EBS-only families.

EBS is a problem because in reality you would be swapping to a network volume. But on a standard VPS server, a bit of swap can really save you some problems. Swap was invented at a time when *NIX was strictly used in a multi-user environment, not far removed from today's webservers but very different from today's consumer devices. It still serves a purpose in that role.

> [...] a bit of swap can really save you some problems.

And cause a bunch more. Have you ever had to debug a race that is only triggered when a process hits swap. Not fun at all.

Re: Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

#99
post #75

Earlier quoted context omitted.

For me swap is an error. I would rather a server crash then go into swap and run slow. If your servers are at 75% CPU and start swapping you are screwed. Time to start adding servers. In this day and age you are wasting a lot of money to be sitting around waiting for swap.

zswap makes swap useful on the server as it results in gradual slowdown when the system has used all the ram. And if the system runs at 75% of CPU, then there is room for such slowdown.

yes, we run ZRAM of every single server we have, its fantastic and we don't have swap anywhere. It's worth noting though that vm.swappiness is set to 0 then ZRAM wont be used, it must be set to 1 or higher unless you're using a really old kernel (pre 3.5).

Re: Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

#100

Earlier quoted context omitted.

IMHO if you can't not use swap without running out of memory in normal workloads, you either need more RAM or to work on reducing your memory consumption; swapping is for temporary overloads or gracefully recovering from one, not a substitute for RAM. It's not just a bunch of people setting cm.swappieness to 0 is it? That behaviour changed and now prevents going into swap /ever/ where it used to mean go into swap as…

I think that the new behavior at zero (documented anywhere?) is a discontinuity. There should be a separate setting for mlocking code pages. Swap is useful even if you have plenty of RAM. It's better to swap out a cold file-backed page and use that page for cache instead than to pin that cold page in memory for no reason.

That's what ZRAM is for.
Post reply on HN