Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Help, Linux ate my RAM

linuxatemyram.com

Post reply on HN
Print view View original

51–60 of 106 posts

Re: Help, Linux ate my RAM

#51

Post by yread »

francoisdevlin wrote: post #24
krobertson wrote: post #19

Earlier quoted context omitted.

I think this is a pretty common misconception overall. Working at startups, often find devs with multiple hats sometimes doing ops tasks. Seen many who hop on a system trying to diagnose some issue, fire up top and proclaim "OMG, the problem is we're running out of memory!" 2nd is explaining virtual/resident set size.

Okay, I'm one of those devs. What is this virtual/resident set size thing you're talking about? EDIT: Thank you for all the helpful responses!

On Windows it's called committed memory

http://blogs.technet.com/b/markrussinovich/archive/2008/11/1...

(the article doesn't have an anchor there... but you know what I mean)

Re: Help, Linux ate my RAM

#52

Post by seclorum »

illumin8 wrote: post #11

I've tried to educate Oracle DBAs on why top is wrong and their memory really isn't being used. It's painful and they often refuse to believe that I know what I'm talking about, and that they should use the free -m command to see what memory is actually available for use. Is there any particular reason why Oracle DBAs are less likely to believe this? Perhaps it's because most of them grew up in legacy UNIX environmen…

Teach them to interpret and understand cat /proc/meminfo .. one of the most interesting things you can do with this is pipe it into gnuplot and watch it over time as things happen. Try it sometimes .. you might get through to one or two.

Re: Help, Linux ate my RAM

#53

Post by CJefferson »

jws wrote: post #6
scott_s wrote: post #5

Earlier quoted context omitted.

People say the same things on Mac: http://news.ycombinator.com/item?id=3584609 It represents a fundamental misunderstanding of how modern OSes work. That misunderstanding is not the problem; modern OSes are complex pieces of software, and most people shouldn't have to understand them. OSes should just work. The problem comes in when people who don't understand how they work get the itch to "improve" their system.

Programmers need to remember that users are like this: (from an Apple discussion thread) … [periodically]'installd' begins using 100% of all 4 CPU cores, my fan goes full speed and the whole computer gets very hot. Seems to happen before Software Update checks for updates. I usually go to the terminal and kill the 'installd' process, which reduces the fan speed and heat to normal within a minute. I wonder if this guy…

I have seen installd peg my CPU to 100% and have killed it. What would you suggest I do?

One of the most annoying features of modern OSes is when some system process just decides to start going wild, eating memory and CPU. Often I find reinstalling is the only way to fix such things.

Re: Help, Linux ate my RAM

#54

Post by scott_s »

maushu wrote: post #40
francoisdevlin wrote: post #24

Earlier quoted context omitted.

Okay, I'm one of those devs. What is this virtual/resident set size thing you're talking about? EDIT: Thank you for all the helpful responses!

I'm a developer too and this is what I understand (might be wrong): - Virtual memory is basically "abstract memory" that is linked (mapped) to RAM or HDD, this means that by accessing this "memory" you might actually be accessing the HDD and, because of this, larger than physical ram. - Virtual set size is the allocated virtual memory (the above) to the process. - Resident set size is the allocated physical (RAM) mem…

Virtual memory is actually more subtle than that.

At a mechanical level, virtual memory is permission from the operating system to use addresses in your address space. It is so called because, as you point out, it allows us to separate the concept of "memory for a process" from "physical memory on a chip." The reason I further refine the concept is that allocating virtual memory does not allocate actual memory. Let's look at an example:

  void* addr = mmap(NULL, 10 * 1024 * 1024, PROT_READ | PROT_WRITE, 
                    MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
That call allocates 10 MB of virtual memory. But there is no physical memory backing any of it. All that has happened is that the operating system has now said "Okay, starting at the address I return to you, you can now access 10 MB of memory. I will do all of the work of making sure physical memory backs the virtual memory when you access it." That is, once I try to access the memory, it will trigger a page fault, and the OS will find a page in physical memory to back my virtual memory. But until that happens, no memory - not in RAM, not on disk - backs the virtual memory.

Re: Help, Linux ate my RAM

#55

Post by scott_s »

jff wrote: post #17
mwexler wrote: post #9

firefoxatemyram.com is still available. Perhaps we can put a site up there as well.

Yep, firefox is currently 555 MB resident and using 1.5 GIGABYTES of virtual memory space. Goddamn, firefox, you are a pig . Saddest thing? I've got gmail, github, and maybe 10 static pages open. Linux just looks like it ate your RAM. Firefox straight up does eat it.

I suspect you don't know what virtual memory is: http://news.ycombinator.com/item?id=3699481

Re: Help, Linux ate my RAM

#56

Post by scott_s »

flomo wrote: post #47
llambda wrote: post #8

Earlier quoted context omitted.

I'm actually continually amazed that even some technically inclined people misunderstand how memory works in /all/ modern OSes: recently I had a discussion on Twitter that went something like this: "OS X is horrible, it uses nearly all of my 8 gigs of RAM and my browser is horribly slow!" to which I replied, "it's perfectly normal for the OS to appropriate the RAM in the fashion you see in Activity Monitor; this does…

On a few occasions, Safari has claimed gigabytes of memory and caused my machine to start thrashing the swap. In that case, the correct answer is to restart Safari, not take a refresher course on virtual memory allocation. Windows Vista had an issue where copying large files would set off such a huge swap-storm that OS became completely unresponsive for several minutes. People gave all the same VM excuses then as wel…

Those aren't excuses, those are reasons: copying large files are the pathological case for just about all caching techniques. You've now kicked out everything useful from your cache with something that you will never use again. It has little to do with virtual memory itself.

Re: Help, Linux ate my RAM

#57

Post by ivan78 »

illumin8 wrote: post #31
ivan78 wrote: post #18

Earlier quoted context omitted.

As a longtime Linux user/admin and beginning Oracle DBA I now know that all memory should be occupied by Oracle, not by OS. :-) Serious mode on: I'm sure it's a big problem to be narrow expert. They can be brilliant specialist in their field, but one step aside and they are absolutely helpless.

Thanks for the great comment. I really wish more people like yourself at least understood Linux memory management. I think it will truly help you to be an exceptional DBA. I can also understand how an expert DBA might not know or care too much about the OS underneath his software, although I would argue that if you understand the OS fundamentals, you will be that much better at whatever specialty you have.

Thank you. I'm on my way to become exceptional DBA. Oracle is only 2 years younger than me, but unfortunately it has been developing much faster than I am. It literally takes years just to become familiar with all these layers of Oracle technologies. So, it's really hard to blame DBAs for not knowing how underlying OS works. :-)

Re: Help, Linux ate my RAM

#58

Post by ArbitraryLimits »

scott_s wrote: post #48
ArbitraryLimits wrote: post #43

Earlier quoted context omitted.

555 MB is still a lot. That's the actual memory usage.

Modern web browser are complicated pieces of software that are also the nexus for most people's interactions with their computer. tiles' point is correct: Firefox will tend to cache things much like the OS itself. We should expect it to use a lot of memory. If your system performance has not suffered, there is no problem.

I'm sorry, but this reply doesn't make any sense to me. If applications have to cache something, they should use the file system, which would not affect their resident memory at all. Maybe I'm just old school, but where I come from operating systems handle the memory hierarchy, not applications.

Re: Help, Linux ate my RAM

#59

Post by click170 »

onli wrote: post #20
jws wrote: post #6

Earlier quoted context omitted.

Programmers need to remember that users are like this: (from an Apple discussion thread) … [periodically]'installd' begins using 100% of all 4 CPU cores, my fan goes full speed and the whole computer gets very hot. Seems to happen before Software Update checks for updates. I usually go to the terminal and kill the 'installd' process, which reduces the fan speed and heat to normal within a minute. I wonder if this guy…

The reaction of the user is not suprising. No background-daemon should use all 4 cores and be that noticable. Users don't feel good when the computer starts to make thing on his own they don't expect. Ubuntu had or has the same issue with a deamon used for the graphical package-administrations-guis rebuilding an index (I forgot the name). They tried to help themselves with appropriate nice-settings, defused the issue…

"You simply can't use the system properly when a background-process uses all ressources. And one normally want do something else than wait for the system. I consider such behaviour a bug."

Agreed, but to add to your point, if your background process is taking up all the resources in my system, that defeats the purpose of being a background process.

Re: Help, Linux ate my RAM

#60

Post by qwe123_troll »

maushu wrote: post #40
francoisdevlin wrote: post #24

Earlier quoted context omitted.

Okay, I'm one of those devs. What is this virtual/resident set size thing you're talking about? EDIT: Thank you for all the helpful responses!

I'm a developer too and this is what I understand (might be wrong): - Virtual memory is basically "abstract memory" that is linked (mapped) to RAM or HDD, this means that by accessing this "memory" you might actually be accessing the HDD and, because of this, larger than physical ram. - Virtual set size is the allocated virtual memory (the above) to the process. - Resident set size is the allocated physical (RAM) mem…

No, not at all.

Virtual memory isn't really memory at all, it's just an entry in a table which marks addresses as accessible to your process. It's up to the OS to decide whether or not these addresses correspond to anything physical or not.

In Linux (and every other modern OS, AFAIK) the OS allocates physical memory on-the-fly, only when your program actually starts reading or writing addresses in the virtual memory table.

Thus, on a 64-bit machine you can allocate thousands of gigabytes of "virtual memory" without using any real memory at all.

Post reply on HN