Diagnosing Memory “Leaks” in Python
chase-seibert.github.io
Diagnosing Memory “Leaks” in Python
1–10 of 32 posts
Re: Diagnosing Memory “Leaks” in Python
#2Kind of surprised the poster didn't know that operating systems often hold on to memory.
> we noticed that the memory of the celery process was continuing to grow.
Doesn't look like there was any bad outcome related to this observation. Was any process not getting the memory it wanted?
Re: Diagnosing Memory “Leaks” in Python
#3Problem seems to be solved in Python 3.3+: http://bugs.python.org/issue11849
The article mentions some interesting disgnostic tools, though.
Re: Diagnosing Memory “Leaks” in Python
#4Problem seems to be solved in Python 3.3+: http://bugs.python.org/issue11849 The article mentions some interesting disgnostic tools, though.
The linked issue probably has nothing to do with the case (see last 3 comments on issue page).
Nice summary of python memory debugging tools though. I would add dowser to the list too.
Re: Diagnosing Memory “Leaks” in Python
#5Nice read, even though I haven't written any Python in a long time. Memory management in modern OSes is very complicated and there isn't a simple answer to "how much memory is my process using exactly". [1] Memory growth / usage isn't usually something to worry about unless memory growth is constant under load and/or OOM killer kicks in.
Also consider that `heapy` will probably only report on objects created by your Python code, and not any memory taken up by native code in the interpreter itself or any shared libraries.
1. For example see:
- http://stackoverflow.com/questions/860878/tracking-actively-...
- https://mail.gnome.org/archives/gnome-list/1999-September/ms...
- http://bmaurer.blogspot.co.uk/2006/03/memory-usage-with-smap...
Re: Diagnosing Memory “Leaks” in Python
#6I encountered the "memory hogging" behavior of Python processes once, where I was sure that my GC worked correctly and that I released all unused objects but still the memory of the process would keep growing. I also remember having this problem with a C++ once as well. It doesn't seem a problem though since the OS should normally release the memory if it's needed by another process. Still, this kind of behavior can definitely drive you nuts.
BTW Celery is usually not a good fit for long running processes, because if you have many of those processes running in parallel within a production system it will get really difficult restarting the Celerey daemon as it will have to wait for all these processes to stop (during which no new tasks can be processed). Why restart at all you ask? Well, restarting is necessary to reload the code, as it is not recommended to use the autoreloader in a production system. This problem persists even when using the multiprocessing module btw (as the author suggested), since on Linux Python uses fork() to create a new process, thereby just copying the whole memory of the given process.
Re: Diagnosing Memory “Leaks” in Python
#7I encountered the "memory hogging" behavior of Python processes once, where I was sure that my GC worked correctly and that I released all unused objects but still the memory of the process would keep growing. I also remember having this problem with a C++ once as well. It doesn't seem a problem though since the OS should normally release the memory if it's needed by another process. Still, this kind of behavior can…
What do you consider as alternative to celery?
Re: Diagnosing Memory “Leaks” in Python
#8I encountered the "memory hogging" behavior of Python processes once, where I was sure that my GC worked correctly and that I released all unused objects but still the memory of the process would keep growing. I also remember having this problem with a C++ once as well. It doesn't seem a problem though since the OS should normally release the memory if it's needed by another process. Still, this kind of behavior can…
What do you consider as alternative to celery?
If you find one, let me know ;) I'm looking for something myself currently.
There's RQ (http://python-rq.org/) but it seems to have a similar design as Celery (just a simpler architecture) so it probably suffers from the same problem.
A good solution would be to have a series of workers that can launch new independent Python processes for each task, e.g. using the subprocess module.
Re: Diagnosing Memory “Leaks” in Python
#9Earlier quoted context omitted.
What do you consider as alternative to celery?
If you find one, let me know ;) I'm looking for something myself currently. There's RQ ( http://python-rq.org/ ) but it seems to have a similar design as Celery (just a simpler architecture) so it probably suffers from the same problem. A good solution would be to have a series of workers that can launch new independent Python processes for each task, e.g. using the subprocess module.
I have long running jobs (say, 5 minutes on average - up to an hour). I originally used celery (after picloud shut down) but it just doesn't work well with those charcteristics. Each worker reserved an extra job so it was impossible to get good cpu utilisation.
I switched to rq and it's all been much easier. The behaviour is easy to understand and it's easy to inspect redis to see what's going on.
In terms of the code restart angle - I'm fairly sure you can effectively restart the workers. They run as a single process that forks to do work. Each copy you run only has a single worker, so you need to run multiple instances yourself. If you kill the parent it waits until the child has finished the job it is on before terminating.
I could be wrong about some of the details. I'd recommend giving it a shot. I must have run 100,000s of jobs through it now and I haven't had a single issue.
Re: Diagnosing Memory “Leaks” in Python
#10import resource
import objgraph
these are some golden modules that I never knew about