Live data from Hacker News

Something is deeply broken in OS X memory management

workstuff.tumblr.com

161–170 of 269 posts

Re: Something is deeply broken in OS X memory management

#161
post #55

What really needs to happen is that Spotlight and Time Machine need to use direct i/o (F_NOCACHE) when they read data from the filesystem, this way they won't pollute the disk cache with their reads and OSX won't swap out a bunch of pages in response. I think you could probably hack something together that does this with DYLD_INSERT_LIBRARIES (OSX's LD_PRELOAD) that would would hook the open system call and fcntl F_N…

For Linux, there is a program called nocache, which does something similar to this:

https://github.com/Feh/nocache

Re: Something is deeply broken in OS X memory management

#162

Earlier quoted context omitted.

Yet it still swaps to disk ALL THE TIME and a new Terminal.app window can take up to 5 seconds to open. That's not swapping. That delay is /usr/bin/login searching the system logs so that it can display the date and time of your last login. Create a .hushlogin file in your home directory to prevent that.

Hard drives can read read data sequentially very, very quickly. On whatever generic SATA hard drive I have in my workstation, I get: eklitzke@gnut:~ $ time sudo head -c 1073741824 /dev/sda > /dev/null real 0m8.267s user 0m0.220s sys 0m0.810s So I can read 1G off the drive in about 8 seconds. Even if the login command does need to sequentially read through logs to find the last login time (and I'm skeptical of that, b…

Log files are not stored sequentially on disk because they are constantly being appended to.

http://en.wikipedia.org/wiki/File_system_fragmentation

Re: Something is deeply broken in OS X memory management

#163
post #148

Earlier quoted context omitted.

Metzger's post is much more subdued and interesting. IMO it would have made a much better submission. Link for the curious (it's linked from this article): https://plus.google.com/116685507294337280246/posts/camYp28M...

I still don't quite grasp it. He is talking about page outs, but how is disabling the dyn-pager helping with that? Shouldn't page outs only happen when RAM is full? At my machine with 8 GB RAM and uptime of 4 days I have page outs of only 2 Mega byte. And page ins of 2 Gigabyte. P.S. I subscribe to your blog! * starstruck

RAM serves (at least) two purposes: holding application data, and caching disk data. Sometimes it can be useful to swap application data out to disk to make more room for disk caching. Imagine if you have an app taking up a whole lot of memory that isn't actively using most of it, and another app reading a lot of data from the disk. In this case, you'll perform better if you swap out all that unused data, and use that RAM to cache disk access for the other app.

What he's saying is happening is that the OS is doing this too aggressively, and that it ends up swapping out data that's actually in use in favor of disk data which doesn't really need to be cached, which hurts performance.

By disabling the pager, you make it impossible to move application data to disk at all. This limits the amount of RAM available for disk caching, but if the OS really is caching too aggressively, that will ensure that it can never page out useful application data by mistake.

My experience mirrors yours, in that it really doesn't seem to be a problem on the computers I've used, but that's what he says he's seeing.

Glad you like the blog, but I'm just a regular guy. I put my pants on with a high speed pants installation robot just like everybody else.

Re: Something is deeply broken in OS X memory management

#164
post #46

Earlier quoted context omitted.

The thing is, Lion runs like crap without an SSD or insane amounts of RAM for many people. It doesn't help if it works for some, or even the majority. If anything, we need more people digging in OS X internals. I am sorry to use a tired analogy here, but Vista did work great for many too. I have two Macs, one brand new, one migrated from SL, and 10.7 Safari was almost unusable on both until I installed SSDs. If that…

I have 12GB of RAM and Lion runs like crap on my iMac 2.93GHz i7. It's like the system discards pages of programs just because the app has been inactive for an hour or so. So when I come back and start the same app the f*cking rotating HD I have sounds like a birds nest for too long periods. Edit: Disabled the pager and the system now seems much more quiet regarding to disk seek noise when I start apps. Feels like a…

> It's like the system discards pages of programs just because the app has been inactive for an hour or so.

From what I've read, Windows memory manager does the same thing - after a while, it swaps out unused pages, even if plenty of free memory is available.

I wonder what's the logic behind this - did the engineers assume that the speedup coming from more free memory being available for disk cache is worth the hassle of waiting for the swapped out page (when it's actually needed)?

Re: Something is deeply broken in OS X memory management

#165

Earlier quoted context omitted.

Doesn't the latter void warranty?

Yes, but you can un-void it by plugging your CD drive back in. Note that replacing the hard drive voids the warranty as well.

Only voids the warranty on the hard drive, in all actuality they don't care... They will simply call up and ask about it and you tell them you replaced it due to corporate policy and all is well.

Re: Something is deeply broken in OS X memory management

#166

I have an 8-way Xeon Mac Pro w/ 20GB of RAM, almost half of which is 'free' at any point during the day unless I'm doing something really out of the ordinary. Yet it still swaps to disk ALL THE TIME and a new Terminal.app window can take up to 5 seconds to open. I really don't give a shit how it's not "technically" broken - that's broken from an experience point of view. And I haven't re-installed the OS (this was an…

I disable swap on all my machines over 4gb... Windows, Linux, or OSX. There are people who will advise against this, but in 5 years I've not had any problems. I think once or twice firefox leaked uncontrollably and therefore killed.

Re: Something is deeply broken in OS X memory management

#167

I have an 8-way Xeon Mac Pro w/ 20GB of RAM, almost half of which is 'free' at any point during the day unless I'm doing something really out of the ordinary. Yet it still swaps to disk ALL THE TIME and a new Terminal.app window can take up to 5 seconds to open. I really don't give a shit how it's not "technically" broken - that's broken from an experience point of view. And I haven't re-installed the OS (this was an…

"Yet it still swaps to disk ALL THE TIME "

You should run some dtrace magic to find out what 'it' is. MIght be the OS, might be a badly behaving application. Who knows.

I find it too easy to blame the OS for all of this. One poorly written app can cause a lot of performance damage.

Re: Something is deeply broken in OS X memory management

#168
post #16

the OS X kernel is open source. so why aren't people reading it to figure out where this bug is?

What are you gonna do about it when you find and fix the bug? You can't run a custom kernel on OS X.

Do what normal people do. Tell Apple:

http://bugreporter.apple.com

Re: Something is deeply broken in OS X memory management

#170

I have an 8-way Xeon Mac Pro w/ 20GB of RAM, almost half of which is 'free' at any point during the day unless I'm doing something really out of the ordinary. Yet it still swaps to disk ALL THE TIME and a new Terminal.app window can take up to 5 seconds to open. I really don't give a shit how it's not "technically" broken - that's broken from an experience point of view. And I haven't re-installed the OS (this was an…

Yet it still swaps to disk ALL THE TIME and a new Terminal.app window can take up to 5 seconds to open. That's not swapping. That delay is /usr/bin/login searching the system logs so that it can display the date and time of your last login. Create a .hushlogin file in your home directory to prevent that.

If you are interested what happens when you open a new terminal window, run 'sudo opensnoop' in one window and then open a new window.

Here the majority of files being opened are in /var/log or Homebrew related.

Also interesting ... creating a .hushlogin did not change much. It still opens about 50 files in /var/log/asl/

Post reply on HN