Now if there were an online course where you could learn this kind of stuff I would be signed up so fast physicists would be re-evaluating general relativity.
We need some online courses dedicated to not-beginners :)
Tracking down a memory leak in Ruby's EventMachine
31–33 of 33 posts
Re: Tracking down a memory leak in Ruby's EventMachine
#32This is slightly off-topic, but I worked on a Ruby project where we did something just like this: "It was easy enough to work around the leak by adding monitoring and restarting the process whenever memory usage grew too large" I was surprised, because I can not think of any other language and/or framework where "just restart the process" is done so often. I mean, this is not a common attitude among Java programmers,…
This isn't uncommon in the python community for long running processes - especially those that frequently that create large amounts of small objects. An application server sitting in front of mongodb does exactly this. It forwards queries to mongo, then pymongo json loads the result for you, the application server does what it wants with the objects, then json dumps them to serve to clients. The json loads/dumps calls create tons of objects and memory bloat. The GC cleans them all up, yet memory isn't returned to the OS. See here:
http://effbot.org/pyfaq/why-doesnt-python-release-the-memory...
In my tests of python 2.6.6, 2.7.3, and 3.3, python 2.7.3 was by far the worst in hanging on to memory. Yet the performance increase (due to integration of simplejson) is worth the memory penalty. We use gunicorn with sync workers to serve our WSGI app and a memory watchdog to signal the worker to gracefully retire after it handles a query large enough to leave it with a large memory footprint.
Performance has been awesome and we're very happy with the result.
Re: Tracking down a memory leak in Ruby's EventMachine
#33Earlier quoted context omitted.
Deduction is great, but what I read here was how much experience matters in debugging. You have to know what to look and test for to provide fodder for your reasoning capabilities, or else you're just shooting in the dark. The OP here had a plan informed by what I'm guessing are years of experience solving this kind of thing ("It's probably small objects. I should dump the core and look for repeating memory patterns.…
Yes, this is true, so as I said, debugging looks easier in hindsight, especially someone else's hindsight :) Summarizing the reasoning process: 1. The program's object space doesn't contain an absurd number of small objects, so inspect the core dump 2. 95% of the core dump is leaked objects, so a random sample should contain clues to the composition of the leaked objects. 3. A repeated pattern in every leaked object…
Yup. Wasn't meaning to disagree with anything you said. Just putting a different emphasis on it than you did.