Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Something is deeply broken in OS X memory management

workstuff.tumblr.com

Post reply on HN
Print view View original

151–160 of 269 posts

Re: Something is deeply broken in OS X memory management

#151

Post by commandar »

Shivetya wrote: post #75
gurkendoktor wrote: post #8

Earlier quoted context omitted.

The thing is, Lion runs like crap without an SSD or insane amounts of RAM for many people. It doesn't help if it works for some, or even the majority. If anything, we need more people digging in OS X internals. I am sorry to use a tired analogy here, but Vista did work great for many too. I have two Macs, one brand new, one migrated from SL, and 10.7 Safari was almost unusable on both until I installed SSDs. If that…

My issue with Lion was that it broke WiFi on my 2009 iMac. Yet the 2011 iMac is just fine with it. There has been a near constant running thread on the Apple support since the day Lion dropped. All sorts of WiFi just going out, sometimes even with the icon being nice enough to gray out. Currently at 145 pages and growing https://discussions.apple.com/thread/3191630?start=2160&... Yeah, the 2009 iMac started working t…

My late 2006 MBP had all sorts of wifi issues with 10.5 the first year or so it was out. Like routine kernel panic-bad level stuff.

Re: Something is deeply broken in OS X memory management

#152

Post by nirvana »

DanI-S wrote: post #56

I'm (fairly) uninformed on this, but from an initial read of the post it seems like memory management may have been optimized for SSD-equipped systems, at the expense of hard disk performance? Whether this is unintentional, part of a calculated tradeoff, or a cynical business/tactical decision is another thing.

1.5TB of spinning rust in my macbook pro, no observations of this problem. Did see something like it when I was using an SSD that was about to fail.

Re: Something is deeply broken in OS X memory management

#153

Post by nirvana »

jhspaybar wrote: post #3

I too have noticed huge issues particularly when using photoshop or final cut pro. I figured it was the applications, but if it's the OS that's definitely a much bigger issue. I regularly restart every 2-3 hours when using those two programs heavily.

In your case, it is the applications. Final Cut Pro X still has some instability, and photoshop is crap.

Re: Something is deeply broken in OS X memory management

#154

Post by zurn »

thought_alarm wrote: post #59
carguy1983 wrote: post #51

I have an 8-way Xeon Mac Pro w/ 20GB of RAM, almost half of which is 'free' at any point during the day unless I'm doing something really out of the ordinary. Yet it still swaps to disk ALL THE TIME and a new Terminal.app window can take up to 5 seconds to open. I really don't give a shit how it's not "technically" broken - that's broken from an experience point of view. And I haven't re-installed the OS (this was an…

Yet it still swaps to disk ALL THE TIME and a new Terminal.app window can take up to 5 seconds to open. That's not swapping. That delay is /usr/bin/login searching the system logs so that it can display the date and time of your last login. Create a .hushlogin file in your home directory to prevent that.

> That delay is /usr/bin/login searching the system logs > so that it can display the date and time of your last login.

Normally Unixes (Linux included) use a pretty efficient binary file called wtmp for that, I'm surprised if OS X doesn't. Reading the last disk block of that file would contain the last login with overwhelming probability.

There has to be a lot of seeks even on a slow laptop rotating hd to get a 5 second delay, with 15 ms seek time you get 333 seeks in 5 secs.

Re: Something is deeply broken in OS X memory management

#155

Post by reddit_clone »

spudlyo wrote: post #55

What really needs to happen is that Spotlight and Time Machine need to use direct i/o (F_NOCACHE) when they read data from the filesystem, this way they won't pollute the disk cache with their reads and OSX won't swap out a bunch of pages in response. I think you could probably hack something together that does this with DYLD_INSERT_LIBRARIES (OSX's LD_PRELOAD) that would would hook the open system call and fcntl F_N…

This is an excellent suggestion that might actually solve whole lot of these issues.

I hope Apple engineers are looking at this thread.

Re: Something is deeply broken in OS X memory management

#156

Post by United857 »

I've gotten a lot of his symptoms, and there might be another cause:

http://reviews.cnet.com/8301-13727_7-20064489-263.html

Bad blocks in the disk, causing the system to beachball frequently due to disk I/O failures when swapping out to disk.

The solution for me was to back up, reformat the disk and zero-ing out everything causing bad sectors to get remapped, and then restore.

Re: Something is deeply broken in OS X memory management

#157

Post by joe_the_user »

w0utert wrote: post #4

Not this again, we already went through this a few weeks ago. Back then, I thought the conclusion was that there is nothing broken about OS X memory management, and that with every 'fix' you come up with, you will just introduce another degenerate corner case. The same holds for any OS, trade-offs are made that may have some negative effect in some cases, to the benefit of the general cases. I don't recognize any of…

Not this again, we already went through this a few weeks ago.

In-fricken-deed...

We have been through this problem again and again and again, in different OSes, at different times and with different things triggering the various problems.

It usually ends with a "neck-beard" saying with enough authority "look, really, they are doing it right even if it seems totally illogical to you and any brokenness is just your configuration, little man". Which is to say "you might not like senseless disk-thrashing but would you rather have your machine randomly freeze when it got out of memory?" And scratching a little more, it comes down to admitting that memory-allocation is a hard problem in its full generality and they don't teach you that in application-programmer-school, and further that the solutions to it that any of these OSes have are tuned-black-magic-split-the-difference-haphazard affairs.

Consider. Either the machine keeps all your information in memory it or keeps on-disk and in-memory and either way, the machine hasn't a clue what information is important to you, my friend. It's just data to it. It's not like the computer is intelligent or anything. Why do you think they call it "random-access-memory"? The problem of dividing up chunks of memory for application programs to use is as hard as dividing up that hard disk for large and small files to live in, EXCEPT that application programs expect to be handed a chunk of contiguous memory when they call malloc. Hard problem even with the powerful tools that have evolved for solving it over the years. So when a given memory management scheme works, it isn't really "fixed", it just has been tuned for the corner-cases that are shouting loudest on the help lines.

And yes indeed, it is "funny" how just getting the "simple stuff" to work is a hard problem. IE you can find lots of simple examples where the standard solution seems fail terribly.

Angry how your 100 GB memory machine isn't faster? Look under the hood and you'll find Scotty from Star Trek shouting "Captain, I'm allocating your memory as fast as I can Sir..."

Re: Something is deeply broken in OS X memory management

#158

Post by spudlyo »

nirvana wrote: post #147
lobster_johnson wrote: post #117

Earlier quoted context omitted.

This is the correct solution. Time Machine and Spotlight should not pollute the OS cache.

I bet the odds are damn good that neither of them ARE polluting the OS Cache. That they are is just speculation from someone whose taken his experience and projected it onto everybody. Since he's a person who mucks around with random system settings (like the one in his article) there's no telling what previous damage he's done to cause this problem.

You might be right. In my testing, mdworker and friends behave like you'd want them to, I don't see them polluting the cache on my Lion machine. Haven't tried time machine yet.

EDIT: I can't get either spotlight or time machine to show any cache polluting behavior at all, at least not in the way that I run them. I used "mdutil -E /" to force a re-index of my disk, and I kicked off an initial time machine backup on a secondary drive I had lying around. I see both backupd and mdworker doing a lot of disk reads using iotop, but top shows my inactive memory not really changing as drastically as I'd expect, like if I were to cat a giant file to /dev/null.

Re: Something is deeply broken in OS X memory management

#160

Post by andyzweb »

tomc1985 wrote: post #48

I did this for awhile and ran into an interesting (if fatal) edge case: I use a 1.5tb external drive formatted in exFAT to minimize cross-platform headaches, and whenever the drive is marked dirty (improper shutdown, eject, etc), OSX will run fsck_exfat on it before I can use it. fsck_exfat isn't a huge deal -- or wouldn't be, if it didn't have a nasty tendency to leak RAM... the moment you plug in, fsck_exfat's foot…

exFAT is Microsoft proprietary shit.
Post reply on HN