Something is deeply broken in OS X memory management • Hacker News Board

Re: Something is deeply broken in OS X memory management

#261

Post by comex » Tue, Apr 24, 2012, 4:20 PM UTC

xpaulbettsx wrote: post #224

daniel02216 wrote: post #211

Earlier quoted context omitted.

Yes you can: http://shantonu.blogspot.com http://objc.id.au/post/10101528120/compiling-the-os-x-10-7-k...

None of these posts detail how to actually replace the kernel, only how to build it, nor what happens when Apple ships an update that changes the kernel.

You install the kernel by running make install, then copying the stuff in BUILD/dst to /. (If you haven't changed any APIs accessible to kexts, you can get away with just copying mach_kernel.) When Apple ships an update, you compile the new kernel.

Re: Something is deeply broken in OS X memory management

#262

Post by mtviewdave » Tue, Apr 24, 2012, 4:23 PM UTC

DrJokepu wrote: post #98

spudlyo wrote: post #77

Earlier quoted context omitted.

The notion that disk cache is so ungodly important that the OS will SWAP MY APPLICATIONS OUT TO DISK PRESERVE IT boggles my mind a bit. Users of desktop systems clearly don't like this behavior, in fact they'll do crazy things like purging disk cache via cron every minute to try to stop this from happening.

Ok, consider the following scenario: Process A (let's call it Safari) allocated 600MB of memory. Out of this 600MB, it hasn't used 400MB for quite a while (because, for example, it contains data for tabs you haven't looked at for hours). Now I'm not sure how Darwin does this but I know for a fact that Windows NT kernels will try to write the contents of in-memory pages to the disk at the first good opportunity; this…

>It's not an unreasonable thing to do on the kernel's part to just drop Safari's 400MB from the physical memory and use it for disk caching Final Cut Pro.

The problem with this line of reasoning is that a large amount of cache will often not give you much more benefit than a small amount. Indeed, that's the nature of caching: you get most of the benefit from the first bit of cache, but the level of added benefit drops dramatically with more cache.

What if using 400MB of cache for FCP only gave 5% of a net performance advantage over using 40MB of cache? Would it still be worth it to take away that extra 360MB from Safari?

And there's the issue of human psychology: people deal much more easily with a little slowdown spread evenly than with a full-on stop for a short amount of time (even if the full-on stop scenario gives you greater average performance). I'd prefer Aperture run 5% more slowly than it might otherwise, if that meant I never saw a beachball when running Safari.

Re: Something is deeply broken in OS X memory management

#263

Post by alexeiz » Tue, Apr 24, 2012, 5:27 PM UTC

lgieron wrote: post #164

Flow wrote: post #46

Earlier quoted context omitted.

I have 12GB of RAM and Lion runs like crap on my iMac 2.93GHz i7. It's like the system discards pages of programs just because the app has been inactive for an hour or so. So when I come back and start the same app the f*cking rotating HD I have sounds like a birds nest for too long periods. Edit: Disabled the pager and the system now seems much more quiet regarding to disk seek noise when I start apps. Feels like a…

> It's like the system discards pages of programs just because the app has been inactive for an hour or so. From what I've read, Windows memory manager does the same thing - after a while, it swaps out unused pages, even if plenty of free memory is available. I wonder what's the logic behind this - did the engineers assume that the speedup coming from more free memory being available for disk cache is worth the hassl…

> From what I've read, Windows memory manager does the same thing - after a while, it swaps out unused pages, even if plenty of free memory is available.

Windows XP does that. It was a common source of grief. I remember it being mentioned as early as 2004. Since Windows Vista the memory manager doesn't have that problem.

Re: Something is deeply broken in OS X memory management

#264

Post by napierzaza » Tue, Apr 24, 2012, 6:10 PM UTC

I can't believe how amazing this fix has been. My cursor is insanely fast now. I was used to deleting text by holding the delete key for some time as it slowly deleted character by character. Now it simply buzzes so fast that I am deleting whole words in an instant. Not to mention terribly laggy CLI response when running simple commands.

I realized that there were maybe some issues, but I didn't think a single command would give so much of a speed gain.

Now if I could only figure out how some of my files revert to previous saves, losing data...

Re: Something is deeply broken in OS X memory management

#265

Post by DrJokepu » Tue, Apr 24, 2012, 6:17 PM UTC

mtviewdave wrote: post #262

DrJokepu wrote: post #98

Earlier quoted context omitted.

Ok, consider the following scenario: Process A (let's call it Safari) allocated 600MB of memory. Out of this 600MB, it hasn't used 400MB for quite a while (because, for example, it contains data for tabs you haven't looked at for hours). Now I'm not sure how Darwin does this but I know for a fact that Windows NT kernels will try to write the contents of in-memory pages to the disk at the first good opportunity; this…

>It's not an unreasonable thing to do on the kernel's part to just drop Safari's 400MB from the physical memory and use it for disk caching Final Cut Pro. The problem with this line of reasoning is that a large amount of cache will often not give you much more benefit than a small amount. Indeed, that's the nature of caching: you get most of the benefit from the first bit of cache, but the level of added benefit drop…

This is a very good point and I think it illustrates well how difficult it is to write a paging / caching system that does the right thing most of the time.

Re: Something is deeply broken in OS X memory management

#266

Post by seanalltogether » Tue, Apr 24, 2012, 6:58 PM UTC

batista wrote: post #256

seanalltogether wrote: post #45

Earlier quoted context omitted.

I had to shut down on a daily basis because of this problem with my system having only 4 GB of ram. There is definitely something broken in there. http://i.imgur.com/ohBEF.png

I don't see anything wrong in this picture. You still have free memory. Even if it didn't show free memory you would probably still be OK, because it would just be reserved for use

In that case it should be marked as inactive memory so it can be reused, that memory that was taken up by the end of the day would never be reclaimed so I was paging a lot.

Re: Something is deeply broken in OS X memory management

#267

Post by joezydeco » Tue, Apr 24, 2012, 8:10 PM UTC

seanalltogether wrote: post #45

obituary_latte wrote: post #22

Earlier quoted context omitted.

>I don't recognize any of his symptoms anyway, and my OS X computers get pretty RAM-heavy use, with almost always a linux VM open, XCode, Safari with ~10 tabs, iTunes with a few thousand songs, etc. Oh boy I wish I could say the same. Admittedly I don't shut down on a daily basis but this didn't used to be a problem in SL. FWIW I've an old tank of a tower that has a video card on its last leg.. shutting down invariab…

I had to shut down on a daily basis because of this problem with my system having only 4 GB of ram. There is definitely something broken in there. http://i.imgur.com/ohBEF.png

And not to change the subject, but why does my instance of Dashboard need 340M of RAM? I have maybe 5-6 widgets. Is this thing spawning WebKit for each one?

Re: Something is deeply broken in OS X memory management

#268

Post by fromhet » Wed, Apr 25, 2012, 1:51 PM UTC

cpeterso wrote: post #214

fromhet wrote: post #25

Earlier quoted context omitted.

I made a bash script for it and I'll share it, hacker 2 hacker. Goes a little something like this: #!/bin/bash purge

What license are you releasing your software under?

Proprietary, nanananana

Re: Something is deeply broken in OS X memory management

#269

Post by batista » Wed, Apr 25, 2012, 5:31 PM UTC

seanalltogether wrote: post #266

batista wrote: post #256

Earlier quoted context omitted.

I don't see anything wrong in this picture. You still have free memory. Even if it didn't show free memory you would probably still be OK, because it would just be reserved for use

In that case it should be marked as inactive memory so it can be reused, that memory that was taken up by the end of the day would never be reclaimed so I was paging a lot.

>In that case it should be marked as inactive memory so it can be reused

That would be the job of the memory manager to decide when to do that. Memory could be kept in the non-free state for longer that it actually is needed, but still be marked internally to be available when needed.

OTOH, if you have paging, as you say, then something is wrong, true.

But I don't think that the screenshot shows something wrong.