Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Performance: Why You Should Almost Always Add Swap Space

haydenjames.io

Post reply on HN
Print view View original

91–100 of 120 posts

Re: Linux Performance: Why You Should Almost Always Add Swap Space

#91

Post by cosarara97 »

tfha wrote: post #29
nisa wrote: post #23

This is dangerous advice. At least on Ubuntu 16.04 with stock kernel running due to memory pressure into swap kills the systems. Basically everything times out and you can only hard-reboot. Without swap you have an OOM at least something you can recover from. If you need swap just add a little like 1-4gb - because if you have spinning disks 64gb swap won't help you at all because every request to swap is magnitudes m…

Not just ubuntu. We tell our engineers to remove their swap partitions because sometimes the test suite will consume all remaining memory and push into swap. If that happens, pretty much the only thing you can do is a hard poweroff. If you're fast, you can Ctrl+C and only lose a minute or two, but often you're just completely stuck. Much better to OOM and have the test suite killed early.

Can ulimit help here?

Re: Linux Performance: Why You Should Almost Always Add Swap Space

#92

Post by detaro »

sevensor wrote: post #81
warrenm wrote: post #75

Earlier quoted context omitted.

Sorry, but you must not be "following the industry" very well. SSDs have lifespans measured in years of constant use. If you think it isn't the case, you need to go back and learn more about those devices you claim to know so much about.

I'm afraid we're misunderstanding each other here. You're talking about the fully-integrated SSD. Not my area of expertise. I'm talking about the endurance of the flash cell transistors. Nobody's claiming significantly higher endurance for the transistors themselves -- what's improved is the stuff on top of them. SSD controllers have gotten better at not trashing any given cell with repeated program / erase cycles. I…

But as a user putting the SSD into my machine I don't care about the low-level cells, I care about what's in the datasheet of the device I bought?

Re: Linux Performance: Why You Should Almost Always Add Swap Space

#93

Post by detaro »

therealmarv wrote: post #41
zzzcpan wrote: post #39

Nowadays linux has this thing called zram disk. It compresses data in memory on the fly. I think general swapping advice would serve most people better if they always add swap on zram first, before swap on ssd or hdd. The difference is that getting data in and out of zram is so fast that you can actually use it as a tier of a slower ram, transparently compressed.

is this activated per default today? Last information I've got from Ubuntu 16.04 times is that it is not activated by default. I know that macOS and Windows do similar to this by default.

Distro-dependent. I think at least some distros for lightweight desktop systems and ChromeOS use it by default, RedHat trusts it enough to officially support it, but I don't think any of the major distros has it on by default.

Re: Linux Performance: Why You Should Almost Always Add Swap Space

#94

Post by sevensor »

detaro wrote: post #92
sevensor wrote: post #81

Earlier quoted context omitted.

I'm afraid we're misunderstanding each other here. You're talking about the fully-integrated SSD. Not my area of expertise. I'm talking about the endurance of the flash cell transistors. Nobody's claiming significantly higher endurance for the transistors themselves -- what's improved is the stuff on top of them. SSD controllers have gotten better at not trashing any given cell with repeated program / erase cycles. I…

But as a user putting the SSD into my machine I don't care about the low-level cells, I care about what's in the datasheet of the device I bought?

Fair point. As with eating sausages, so with using flash memory. Maybe easier to make a rational decision about it if you haven't seen it made.

Re: Linux Performance: Why You Should Almost Always Add Swap Space

#95

Post by mmjaa »

jacquesm wrote: post #80
cat199 wrote: post #79

Earlier quoted context omitted.

if swap is 'cargo cult administration', pretending it won't save you a few times even with all the safeguards you claim that you perfectly do is 'narcissist administration'. if your systems are so advanced that you've done all of this up front profiling, you'd know why the system in the pipe decided 100ms is as good as 10 because you'd be monitoring and alerting on memory/swap and would get the message when it hits 8…

If you're architecting your systems in such a way that there is a dynamic component that can cause you to hit the swap then your could also simply buy that much more memory so it won't save you. All you need to do is to monitor your memory usage, any growth that you do not understand is a reason to stop what you are doing immediately and to figure out what is going on. A few GB extra swap space will not save your bac…

There have been many times, in 30 years, where I've been glad I've got swap enabled so I can recover a nearly-out-of-control production system in time, with the right procedures, instead of everything just being killed.

Thats enough for me to turn it on. But, also, gigs of RAM is another must-do cargo-cult thing...

Re: Linux Performance: Why You Should Almost Always Add Swap Space

#96

Post by mmjaa »

LinuxBender wrote: post #21

If you must use swap (you probably don't need to), then at least set zswap.enabled=1 in the kernel boot options, typically in grub. This will enable lzo compression of swap in memory so there is less writing to disk. Some newer kernels use lz4. Either way, what most folks are actually missing is the correct kernel settings for the amount of memory they have. Sadly, the kernel does not dynamically adjust these based o…

Those 3 sync's in rapid succession are secret-monkey-code for "tell the tape unit to rewind..", just FYI .. you don't strictly need 3. ;)

Re: Linux Performance: Why You Should Almost Always Add Swap Space

#97

Post by mnw21cam »

borman wrote: post #85
mnw21cam wrote: post #37

Swap engineering is definitely way more complicated than the article lets on. What people often don't realise is that Linux distinguishes between dirty and clean memory, with dirty memory not reflected in swap, and clean memory already stored somewhere on disc, and program code is counted as clean memory that just happens to live somewhere other than swap . Therefore, under memory pressure (especially if you set swap…

> Therefore, under memory pressure (especially if you set swappiness to zero), you will be preferentially swapping out your program code (because it is always clean) in preference to your program data. Regarding systems with near 100% utilization, wouldn't it be a better advice to pin all executable code to memory via tmpfs (at system level) or mlockall() (application level)? Encountered this case on some heavily-loa…

I'd love to have a configuration option on Linux that elevates the penalty for evicting executable pages, or just makes them non-discardable. I think it would help a lot.

Re: Linux Performance: Why You Should Almost Always Add Swap Space

#98

Post by jacquesm »

mmjaa wrote: post #95
jacquesm wrote: post #80

Earlier quoted context omitted.

If you're architecting your systems in such a way that there is a dynamic component that can cause you to hit the swap then your could also simply buy that much more memory so it won't save you. All you need to do is to monitor your memory usage, any growth that you do not understand is a reason to stop what you are doing immediately and to figure out what is going on. A few GB extra swap space will not save your bac…

There have been many times, in 30 years, where I've been glad I've got swap enabled so I can recover a nearly-out-of-control production system in time, with the right procedures, instead of everything just being killed. Thats enough for me to turn it on. But, also, gigs of RAM is another must-do cargo-cult thing...

Did you do root cause analysis and did you make sure that same condition could never happen again or did you figure that since the swap file saved you that no further action was needed?

The 'many times' has me worried.

Re: Linux Performance: Why You Should Almost Always Add Swap Space

#99

Post by tinus_hn »

detaro wrote: post #92
sevensor wrote: post #81

Earlier quoted context omitted.

I'm afraid we're misunderstanding each other here. You're talking about the fully-integrated SSD. Not my area of expertise. I'm talking about the endurance of the flash cell transistors. Nobody's claiming significantly higher endurance for the transistors themselves -- what's improved is the stuff on top of them. SSD controllers have gotten better at not trashing any given cell with repeated program / erase cycles. I…

But as a user putting the SSD into my machine I don't care about the low-level cells, I care about what's in the datasheet of the device I bought?

The magic of the SSD controller can introduce rather surprising failure modes that you should probably care about.

Re: Linux Performance: Why You Should Almost Always Add Swap Space

#100

Post by AstralStorm »

cosarara97 wrote: post #91
tfha wrote: post #29

Earlier quoted context omitted.

Not just ubuntu. We tell our engineers to remove their swap partitions because sometimes the test suite will consume all remaining memory and push into swap. If that happens, pretty much the only thing you can do is a hard poweroff. If you're fast, you can Ctrl+C and only lose a minute or two, but often you're just completely stuck. Much better to OOM and have the test suite killed early.

Can ulimit help here?

Ulimit is per process, it won't do. vm.overcommit_memory=2 with some setting of vm.overcommit_ratio or bytes will help though.

I'd say overcommit heuristics break applications and cause them to eat too much memory since they don't know when to stop. The only trouble is KVM which for some reason takes double of process space allocated to the VM for no good reason and perhaps memory intensive Java.

You need to tune the latter anyway.

Post reply on HN