Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Help, Linux ate my RAM

linuxatemyram.com

Post reply on HN
Print view View original

71–80 of 106 posts

Re: Help, Linux ate my RAM

#71

Post by ivan78 »

onli wrote: post #20
jws wrote: post #6

Earlier quoted context omitted.

Programmers need to remember that users are like this: (from an Apple discussion thread) … [periodically]'installd' begins using 100% of all 4 CPU cores, my fan goes full speed and the whole computer gets very hot. Seems to happen before Software Update checks for updates. I usually go to the terminal and kill the 'installd' process, which reduces the fan speed and heat to normal within a minute. I wonder if this guy…

The reaction of the user is not suprising. No background-daemon should use all 4 cores and be that noticable. Users don't feel good when the computer starts to make thing on his own they don't expect. Ubuntu had or has the same issue with a deamon used for the graphical package-administrations-guis rebuilding an index (I forgot the name). They tried to help themselves with appropriate nice-settings, defused the issue…

You are probably talking about apt-xapian-index. It rebuilds its indexes once a week, consuming all system resources. I had an old notebook, which became unusable every weekend. I did not want to buy a new machine, so I simply deinstalled apt-xapian-index package. :-)

Re: Help, Linux ate my RAM

#72

Post by ArbitraryLimits »

scott_s wrote: post #64
ArbitraryLimits wrote: post #58

Earlier quoted context omitted.

I'm sorry, but this reply doesn't make any sense to me. If applications have to cache something, they should use the file system, which would not affect their resident memory at all. Maybe I'm just old school, but where I come from operating systems handle the memory hierarchy, not applications.

They cache it in memory to avoid latency. Writing to and reading from disk increases latency. Even crossing the kernel boundary will increase latency. Modern web browsers have architectures similar to OSes at this point - because they have requirements similar to OSes. I think it's natural that they will take on some of the same responsibilities.

Where I come from, what you're describing is called pre-fetching, not caching. That's why your earlier comment confused me.

Re: Help, Linux ate my RAM

#73

Post by sciurus »

Legion wrote: post #38
krobertson wrote: post #19

Earlier quoted context omitted.

I think this is a pretty common misconception overall. Working at startups, often find devs with multiple hats sometimes doing ops tasks. Seen many who hop on a system trying to diagnose some issue, fire up top and proclaim "OMG, the problem is we're running out of memory!" 2nd is explaining virtual/resident set size.

2 minute fix: get people to use htop, not top.

Having tried both, I'd recommend atop over htop. There's a great overview of its capabilities at https://lwn.net/Articles/387202/

Re: Help, Linux ate my RAM

#74

Post by scott_s »

ArbitraryLimits wrote: post #72
scott_s wrote: post #64

Earlier quoted context omitted.

They cache it in memory to avoid latency. Writing to and reading from disk increases latency. Even crossing the kernel boundary will increase latency. Modern web browsers have architectures similar to OSes at this point - because they have requirements similar to OSes. I think it's natural that they will take on some of the same responsibilities.

Where I come from, what you're describing is called pre-fetching, not caching. That's why your earlier comment confused me.

No, I'm talking about caching: keeping data around that you have previously used, assuming you will use it again. However, in order to support caching, you must pre-allocate memory.

Re: Help, Linux ate my RAM

#75

Post by adavies42 »

illumin8 wrote: post #34
francoisdevlin wrote: post #24

Earlier quoted context omitted.

Okay, I'm one of those devs. What is this virtual/resident set size thing you're talking about? EDIT: Thank you for all the helpful responses!

Short and overly simple answer: If a program allocates 2GB of RAM, it might not need to use it all right away. The kernel pretends like it just gave the program 2GB of RAM, but it doesn't really need to flush all those unused pages of cache yet. It can still use them for other things like disk cache, and when the program tries to access those pages then the kernel will dynamically page them in and out as needed. This…

then there's over-commit, e.g. "you want to malloc 200% of the ram? here you go! enjoy! oh wait, you wrote to the second half? have fun finding the power button, sucker!"

can someone explain to me why so many distros, including "enterprise" server stuff, ship with /proc/sys/vm/overcommit_memory set to 1?

Re: Help, Linux ate my RAM

#77

Post by regularfry »

scott_s wrote: post #64
ArbitraryLimits wrote: post #58

Earlier quoted context omitted.

I'm sorry, but this reply doesn't make any sense to me. If applications have to cache something, they should use the file system, which would not affect their resident memory at all. Maybe I'm just old school, but where I come from operating systems handle the memory hierarchy, not applications.

They cache it in memory to avoid latency. Writing to and reading from disk increases latency. Even crossing the kernel boundary will increase latency. Modern web browsers have architectures similar to OSes at this point - because they have requirements similar to OSes. I think it's natural that they will take on some of the same responsibilities.

If it's cached to the filesystem, it'll be handled by the file cache, under the kernel's control with the rest of RAM. Maybe browser makers have a good reason for thinking they can do better than the OS, I don't know, but having two systems trying to do the same job with the same resources sounds like a recipe for instability and inefficiency to me.

Re: Help, Linux ate my RAM

#78

Post by scott_s »

regularfry wrote: post #77
scott_s wrote: post #64

Earlier quoted context omitted.

They cache it in memory to avoid latency. Writing to and reading from disk increases latency. Even crossing the kernel boundary will increase latency. Modern web browsers have architectures similar to OSes at this point - because they have requirements similar to OSes. I think it's natural that they will take on some of the same responsibilities.

If it's cached to the filesystem, it'll be handled by the file cache, under the kernel's control with the rest of RAM. Maybe browser makers have a good reason for thinking they can do better than the OS, I don't know, but having two systems trying to do the same job with the same resources sounds like a recipe for instability and inefficiency to me.

If you have to cross the kernel boundary every time you want to access something in your cache, your "cache" is now much, much slower. Note that this applies even if the OS keeps the file in memory, and doesn't require going to disk.

Re: Help, Linux ate my RAM

#79

Post by philsnow »

jws wrote: post #69
CJefferson wrote: post #53

Earlier quoted context omitted.

I have seen installd peg my CPU to 100% and have killed it. What would you suggest I do? One of the most annoying features of modern OSes is when some system process just decides to start going wild, eating memory and CPU. Often I find reinstalling is the only way to fix such things.

Let it finish. It has some thinking to do, maybe crypto checksums? Maybe just a really inefficient corner case on an algorithm. There's nothing wrong with letting your CPU work. if running at 100% makes your machine flakey, it is broken. If your foreground performance is being impacted too severely (and I haven't seen this from installd, I just noticed and researched installd while removing Mac Keeper (malware) from…

"then reboot. It's extreme, but it has the best chance of getting your processes shut down cleanly as opposed to a kill"

so... how do reboots work on OS X ? on every *nix flavor I know, there's one command that just halts the damn machine and damn the torpedoes, and there's one command that does it more gracefully, by sending progressively harder-to-ignore signals to ~every process except init, ending up on SIGKILL (which is not trappable).

Re: Help, Linux ate my RAM

#80

Post by spudlyo »

JadeRobbins wrote: post #46
Legion wrote: post #38

Earlier quoted context omitted.

2 minute fix: get people to use htop, not top.

Thanks! htop is MUCH better than plain ol top.

You'd be surprised how good and feature rich plain ole top actually is. My guess is you rarely deviate from the default options. I made a video which shows how you can get top to show you some pretty interesting stuff.

http://www.youtube.com/watch?v=yFKRsLj_Jhg

Post reply on HN