Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Help, Linux ate my RAM

linuxatemyram.com

Post reply on HN
Print view View original

81–90 of 106 posts

Re: Help, Linux ate my RAM

#81

Post by onemoreact »

henrikschroder wrote: post #61
ivan78 wrote: post #50

Earlier quoted context omitted.

Why downvoting? It may be not the best explanation, but IMO the closest to the way a "developer" should understand it. On Linux a virtual memory is committed with mmap call. Check this for details: http://stackoverflow.com/questions/2782628/any-way-to-reserv... This is basically how shared libraries and other shared objects are attached to your process.

> Why downvoting? > you must read a book if you want to understand it. "Go read a book" is not a very constructive or helpful answer to a question that - as it turns out - could be answered with a brief comment.

It can be when a short description completely and accurately describes what's going on, but the person creating the comment needed to read a few books on the subject to actually understand both what's happening and why due to other issues.

EX: How does virtual memory get mapped to L1 cache?

Re: Help, Linux ate my RAM

#82

Post by calloc »

ineedtosleep wrote: post #49
jff wrote: post #17

Earlier quoted context omitted.

Yep, firefox is currently 555 MB resident and using 1.5 GIGABYTES of virtual memory space. Goddamn, firefox, you are a pig . Saddest thing? I've got gmail, github, and maybe 10 static pages open. Linux just looks like it ate your RAM. Firefox straight up does eat it.

Why is Firefox always mentioned in this context? Sure they have had a history of it, but lately they've been fine and if one compares the combined spawned processes of Chrome, Chrome typically has more memory consumption.

Remember that when you are accounting for those you also need to remove all of the stuff that is paged into every process and is the same across them, such as shared libraries and the like. Most likely Chrome is using less ram than you think.

Re: Help, Linux ate my RAM

#83

Post by spudlyo »

dfc wrote: post #68
click170 wrote: post #4

It's cute that this pops up every few years, and I think it points to a steady (if slow) attraction of new users to Linux.

It makes me feel old. As soon as I saw the headline I knew what the story was going to be. Another fresh wave of young faces learning about memory...I just wish they'd stay off the lawn.

It makes me feel a little embarrassed for HN actually. This article might be a revelation for the noobs on the Linux subreddit, but I'd expect the HN crowd to find it pretty pedestrian.

Not only that, but in some cases it is flat-out wrong:

No, disk caching only borrows the ram that applications don't currently want. It will not use swap. If applications want more memory, they just take it back from the disk cache. They will not start swapping.

Try again. You can tune this to some extent with /proc/sys/vm/swappiness but Linux is loathe to abandon buffer cache, and will often choose to swap old pages instead.

I have learned this the hard way. For example, on a database machine (where > 80% of the memory is allocated to the DB's buffer pool) try to take a consistent filesystem snapshot of the db's data directory and then rsync it to another machine. The rsync process will read a ton of data, and Linux will dutifully (and needlessly) try to jam this into the already full buffer cache. Instead of ejecting the current contents of the buffer cache, Linux will madly start swapping out database pages trying to preserve buffer cache.

Some versions of rsync support direct i/o on read to avoid this, but they're not mainstream and readily available on Linux. You can also use iflag=direct with dd to get around this problem.

Re: Help, Linux ate my RAM

#84

Post by Moto7451 »

philsnow wrote: post #79
jws wrote: post #69

Earlier quoted context omitted.

Let it finish. It has some thinking to do, maybe crypto checksums? Maybe just a really inefficient corner case on an algorithm. There's nothing wrong with letting your CPU work. if running at 100% makes your machine flakey, it is broken. If your foreground performance is being impacted too severely (and I haven't seen this from installd, I just noticed and researched installd while removing Mac Keeper (malware) from…

"then reboot. It's extreme, but it has the best chance of getting your processes shut down cleanly as opposed to a kill" so... how do reboots work on OS X ? on every *nix flavor I know, there's one command that just halts the damn machine and damn the torpedoes, and there's one command that does it more gracefully, by sending progressively harder-to-ignore signals to ~every process except init, ending up on SIGKILL (…

A standard reboot from the GUI should be fine. If you boot up in verbose mode there are some amusing messages that are displayed at the top of the screen during reboot/shut down when a program has to be force quit by the OS.

Re: Help, Linux ate my RAM

#85

Post by ez77 »

I found this page because I was pretty confused about these issues, and still am...

Question for the crowd: In this site the example given says that in reality there are 869MB of used RAM. I'm comparing this with my VPS values, and would like to know if this is the sum of some column in top. Is it? It looks like it's pretty close to the sum of the SHR column. Does this make sense? Thanks in advance.

Re: Help, Linux ate my RAM

#86

Post by tedunangst »

scott_s wrote: post #48
ArbitraryLimits wrote: post #43

Earlier quoted context omitted.

555 MB is still a lot. That's the actual memory usage.

Modern web browser are complicated pieces of software that are also the nexus for most people's interactions with their computer. tiles' point is correct: Firefox will tend to cache things much like the OS itself. We should expect it to use a lot of memory. If your system performance has not suffered, there is no problem.

Performance does suffer, because the operating system doesn't know that all that memory being used by firefox is just cache. And firefox doesn't know when the operating system would like to use memory for something other than cache. So when you start a second large program, it's thrashing time.

VMs like vmware have the exact same problem, where the host might want memory used by the guest, and you end up with weird scenarios where the guest's swap is in the host's disk cache, but the guest's memory is in the host's swap. One of the things guest tools are supposed to do is communicate with the host regarding memory pressure. Firefox lacks this feedback mechanism.

Re: Help, Linux ate my RAM

#87

Post by aliguori »

adavies42 wrote: post #75
illumin8 wrote: post #34

Earlier quoted context omitted.

Short and overly simple answer: If a program allocates 2GB of RAM, it might not need to use it all right away. The kernel pretends like it just gave the program 2GB of RAM, but it doesn't really need to flush all those unused pages of cache yet. It can still use them for other things like disk cache, and when the program tries to access those pages then the kernel will dynamically page them in and out as needed. This…

then there's over-commit, e.g. "you want to malloc 200% of the ram? here you go! enjoy! oh wait, you wrote to the second half? have fun finding the power button, sucker!" can someone explain to me why so many distros, including "enterprise" server stuff, ship with /proc/sys/vm/overcommit_memory set to 1?

Most software cannot cope with malloc failing. A little swapping is at least recoverable. A failed malloc almost guarantees that your application will crash--sometimes in a very unpleasant way.

Re: Help, Linux ate my RAM

#88

Post by orthecreedence »

pcwalton wrote: post #41
jff wrote: post #17

Earlier quoted context omitted.

Yep, firefox is currently 555 MB resident and using 1.5 GIGABYTES of virtual memory space. Goddamn, firefox, you are a pig . Saddest thing? I've got gmail, github, and maybe 10 static pages open. Linux just looks like it ate your RAM. Firefox straight up does eat it.

What does about:memory say?

I never knew this existed...thanks for pointing this out.

Re: Help, Linux ate my RAM

#89

Post by jff »

scott_s wrote: post #55
jff wrote: post #17

Earlier quoted context omitted.

Yep, firefox is currently 555 MB resident and using 1.5 GIGABYTES of virtual memory space. Goddamn, firefox, you are a pig . Saddest thing? I've got gmail, github, and maybe 10 static pages open. Linux just looks like it ate your RAM. Firefox straight up does eat it.

I suspect you don't know what virtual memory is: http://news.ycombinator.com/item?id=3699481

Yes, I know what virtual memory is, I'm a computer engineer and I work on operating systems. I'm just extremely impressed that they needed 1.5 GB of address space, for christ's sake.

Insert here some pithy comment about Apollo missions and Twitter, or whatever.

Re: Help, Linux ate my RAM

#90

Post by jff »

pcwalton wrote: post #41
jff wrote: post #17

Earlier quoted context omitted.

Yep, firefox is currently 555 MB resident and using 1.5 GIGABYTES of virtual memory space. Goddamn, firefox, you are a pig . Saddest thing? I've got gmail, github, and maybe 10 static pages open. Linux just looks like it ate your RAM. Firefox straight up does eat it.

What does about:memory say?

Basically the same thing. It's been several hours, right now it says it has 644 MB of explicit allocations, it is 675 MB resident, and has 1,633 MB of virtual address space. When I looked at about:memory right after composing my original post, the numbers were within a few megabytes of what htop reported.
Post reply on HN