Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

linuxatemyram.com

Post reply on HN
Print view View original

61–70 of 74 posts

Re: Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

#61

Post by wazoox »

jerf wrote: post #54
rufugee wrote: post #45

Earlier quoted context omitted.

Care to explain why this is so?

Stop measuring your swap space in terms of disk space, start imagining it in terms of "amount of time it takes to fill up". At a nice round 20MB/sec write, it would take a solid 400 seconds to use that much swap at full speed. In reality, you can't fill it that fast either, because it's seeking, also yanking other stuff out to run your other processes, and then sticking the stuff back in to run those other processes,…

Please note that this doesn't apply all the same to servers. I have servers running with 16 GB of RAM and 16 GB of swap (because I simply can't even stop them to add RAM), and even then about twice a year they run out of memory and the OOM killer does its dirty job. However without swap they used to simply crash and burn, so swap is fine in this case.

Re: Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

#63

Post by Andys »

stcredzero wrote: post #59

Hmm, this situation is like owning an electric vehicle. The manufacturers are passionate about technology, and they invented a flywheel that can store massive amounts of energy, giving your vehicle incredible transient acceleration response. They ship many vehicles with this new feature, but unfortunately, they forgot to account for what this would do to the battery charge display after starting. As a result, their r…

If only electric cars were free like Linux.

Re: Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

#64

Post by Luyt »

eliben wrote: post #2

The title of the link (unlike the linked page) suggests that Linux is somehow to blame here... Why? Using RAM for disk caching is a good thing . Besides, this technique is employed by Windows as well. It's just that the average Windows user probably doesn't know how much free RAM his PC has (well, the really average user doesn't know how much total RAM it has). My Windows 7 box takes at least 2GB of memory to itself…

I think the average user doesn't even know what RAM is. My neighbour, for example, doesn't know the distinction between the space available in RAM and the capacity of his harddisk. Which led to a strange misunderstanding when he told me his PC had '250 giga of internal memory'. I tried to explain than with 'internal memory' people usually mean something else, but I didn't think he understood. Never mind, he doesn't grasp hierarchical filesystems either, but all is fine as long as he can edit his photos (which in his perception are kept within Elements) and 'surf the internet'.

Re: Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

#65

Post by spudlyo »

You can't disable disk caching. The only reason anyone ever wants to disable disk caching is because they think it takes memory away from their applications, which it doesn't! Disk cache makes applications load faster and run smoother, but it NEVER EVER takes memory away from them! Therefore, there's absolutely no reason to disable it!

Never say never. You can and should disable/purge linux buffer caching in certain situations. See O_DIRECT and/or POSIX_FADV_DONTNEED. If I'm writing a database, I take care of my own caching, I do not need the OS doing it as well. Also, if I'm doing a huge rsync of a directory tree, just because I just read every byte of every file in that tree, I most certainly do not want to load all of that crap into the linux buffer cache. It will evict my already warm set, as well as potentially cause a ton of swapping.

Re: Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

#66

Post by cookiecaper »

jerf wrote: post #47
cookiecaper wrote: post #44

Earlier quoted context omitted.

Hibernation requires a large swap space, doesn't it? Last I knew it did, perhaps you can compress it now.

Yes, and it may seem a merely semantic difference, but there's still a difference between "swap space" and "hibernation backing". One I'm willing to wait for while it fills, the other, I am not. The kernel may not distinguish, but I do. When I don't care about hibernation I just remove it.

Right, swapoff is totally acceptable imo if you're having trouble with swap. I was commenting on how "double the size of RAM" is a bad or outdated guideline. There's been a few times where I've regretted not making my swap space big enough, sometimes when getting a RAM upgrade and wanting to do hibernation, etc. My disk isn't super pressed for space so in my mind there's no real reason to be stingy, and then you don't have to grow/shrink partitions if an upgrade occurs after the initial disk setup.

Re: Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

#67

Post by jerf »

wazoox wrote: post #61
jerf wrote: post #54

Earlier quoted context omitted.

Stop measuring your swap space in terms of disk space, start imagining it in terms of "amount of time it takes to fill up". At a nice round 20MB/sec write, it would take a solid 400 seconds to use that much swap at full speed. In reality, you can't fill it that fast either, because it's seeking, also yanking other stuff out to run your other processes, and then sticking the stuff back in to run those other processes,…

Please note that this doesn't apply all the same to servers. I have servers running with 16 GB of RAM and 16 GB of swap (because I simply can't even stop them to add RAM), and even then about twice a year they run out of memory and the OOM killer does its dirty job. However without swap they used to simply crash and burn, so swap is fine in this case.

The only thing I can possibly imagine that a server can do with 16GB of swap is leaking garbage like a sieve and then getting the garbage swapped out. On an incredibly local basis, yes, this might be a better idea than not having swap, but as a generalized reason for servers to have swap it's terrible. If that's not what's happening I am all ears as to what the situation really is, but I'm sure it'll be something very unusual; in general if your server has even the slightest need for performance it can't use swap.

Re: Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

#68

Post by jacques_chester »

colanderman wrote: post #21

This is misleading, as the kernel will often swap out processes rather than shrink the disk cache. Its tendency to do this is controlled by the vm.swappiness parameter, which by default heavily favors swapping: http://kerneltrap.org/node/3000 Decreasing this value will favor reducing the cache size and will tend to increase responsiveness. You can set it in the /etc/sysctl.conf file. Low values (≤ 5) are especially g…

Low swappiness is also useful in tightly controlled VPSes. Swap is absolute murder on a VPS platform. I leave it on as emergency backstop, but otherwise I have turned down to 0. I simply do not want processes swapped out, ever.

Re: Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

#69

Post by stcredzero »

Andys wrote: post #63
stcredzero wrote: post #59

Hmm, this situation is like owning an electric vehicle. The manufacturers are passionate about technology, and they invented a flywheel that can store massive amounts of energy, giving your vehicle incredible transient acceleration response. They ship many vehicles with this new feature, but unfortunately, they forgot to account for what this would do to the battery charge display after starting. As a result, their r…

If only electric cars were free like Linux.

My analogy has more to do with approaches to UX than it does with economic model.

Re: Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

#70

Post by js2 »

spudlyo wrote: post #65

You can't disable disk caching. The only reason anyone ever wants to disable disk caching is because they think it takes memory away from their applications, which it doesn't! Disk cache makes applications load faster and run smoother, but it NEVER EVER takes memory away from them! Therefore, there's absolutely no reason to disable it! Never say never. You can and should disable/purge linux buffer caching in certain…

Funny, I just noticed this in the Mac OS X rsync man page for the very first time:

       --cache
              Apple specific option to enable filesystem caching of rsync file
              i/o Otherwise fcntl(F_NOCACHE) is used to limit memory growth.
Post reply on HN