Diagnosing Memory “Leaks” in Python

Post by falsedan » Tue, Sep 08, 2015, 6:26 AM UTC

ldarby wrote: post #13

cheez wrote: post #2

Kind of surprised the poster didn't know that operating systems often hold on to memory. > we noticed that the memory of the celery process was continuing to grow. Doesn't look like there was any bad outcome related to this observation. Was any process not getting the memory it wanted?

There are in fact bad outcomes of continued memory growth - firstly there is a catastrophic performance degradation when processes start getting swapped to disk, then when swap space runs out Linux's OOM killer starts killing processes in an attempt to free up memory. Kind of surprised you didn't know this.

Post by toong » Tue, Sep 08, 2015, 7:33 AM UTC

danudey wrote: post #19

IgorPartola wrote: post #17

Once I had to fix a severe memory leak in piece of Python code. None of the available tools revealed anything useful. I ended up just adding debug statements to the hot loop to see which step caused the memory usage to jump. Should have known the answer: a bad C extension was leaking memory. Lesson learned.

libzmq apparently has a bug where it creates structures to handle connections every time you connect, and it cleans them up after processing data. This means that if you connect then disconnect without sending data, you leak memory. It's trivial to run a "for i in …" and cause a zmq app to leak gigabytes of memory in a matter of seconds. Highly problematic.

Post by ThePhysicist » Tue, Sep 08, 2015, 8:14 AM UTC

aidos wrote: post #9

ThePhysicist wrote: post #8

Earlier quoted context omitted.

If you find one, let me know ;) I'm looking for something myself currently. There's RQ ( http://python-rq.org/ ) but it seems to have a similar design as Celery (just a simpler architecture) so it probably suffers from the same problem. A good solution would be to have a series of workers that can launch new independent Python processes for each task, e.g. using the subprocess module.

I have long running jobs (say, 5 minutes on average - up to an hour). I originally used celery (after picloud shut down) but it just doesn't work well with those charcteristics. Each worker reserved an extra job so it was impossible to get good cpu utilisation. I switched to rq and it's all been much easier. The behaviour is easy to understand and it's easy to inspect redis to see what's going on. In terms of the cod…

Post by [deleted] » Tue, Sep 08, 2015, 8:17 AM UTC

polpo wrote: post #18

ThePhysicist wrote: post #6

I encountered the "memory hogging" behavior of Python processes once, where I was sure that my GC worked correctly and that I released all unused objects but still the memory of the process would keep growing. I also remember having this problem with a C++ once as well. It doesn't seem a problem though since the OS should normally release the memory if it's needed by another process. Still, this kind of behavior can…

We solved the Celery + long running + code reloading problem by having each push of new code be associated with a new Celery queue. On push of new code, start the new queue, and SIGTERM the old one, which will wait for any long-running tasks to finish before exiting.

Post by ldarby » Tue, Sep 08, 2015, 1:12 PM UTC

cheez wrote: post #22

ldarby wrote: post #13

Earlier quoted context omitted.

There are in fact bad outcomes of continued memory growth - firstly there is a catastrophic performance degradation when processes start getting swapped to disk, then when swap space runs out Linux's OOM killer starts killing processes in an attempt to free up memory. Kind of surprised you didn't know this.

Except that didn't happen, according to the post.