Fixing a Tough Memory Leak in Python
info.cloudquant.com
Fixing a Tough Memory Leak in Python
1–10 of 15 posts
Re: Fixing a Tough Memory Leak in Python
#2When I worked on a Python app that did real-time processing of (very large) lists of python dicts, merely processing them in a list comprehension caused "leaks" causing unbound virtual memory growth. The issue was that that for small allocations, eg. interned strings, malloc would use the "brk" call rather than "mmap". "brk" simply bumps the top of heap pointer, so if a more long-lived object gets allocated during the time of the comprehension it creates high-water marks in the heap.
The solution at the time was to process the large lists in smaller chunks at a time to make it less likely that a small longer lived object would be allocated in the middle, preventing later cleanup. However, a better solution would probably have been to rewrite parts of the app in a less allocation-happy language.
Re: Fixing a Tough Memory Leak in Python
#3When I worked on a Python app that did real-time processing of (very large) lists of python dicts, merely processing them in a list comprehension caused "leaks" causing unbound virtual memory growth. The issue was that that for small allocations, eg. interned strings, malloc would use the "brk" call rather than "mmap". "brk" simply bumps the top of heap pointer, so if a more long-lived object gets allocated during th…
Python is not efficient, but the bug you described seems like a problem with that implementation of malloc...
Re: Fixing a Tough Memory Leak in Python
#4When I worked on a Python app that did real-time processing of (very large) lists of python dicts, merely processing them in a list comprehension caused "leaks" causing unbound virtual memory growth. The issue was that that for small allocations, eg. interned strings, malloc would use the "brk" call rather than "mmap". "brk" simply bumps the top of heap pointer, so if a more long-lived object gets allocated during th…
Can you construct a simple example case of that? I think you should report that upstream.
Re: Fixing a Tough Memory Leak in Python
#5When I worked on a Python app that did real-time processing of (very large) lists of python dicts, merely processing them in a list comprehension caused "leaks" causing unbound virtual memory growth. The issue was that that for small allocations, eg. interned strings, malloc would use the "brk" call rather than "mmap". "brk" simply bumps the top of heap pointer, so if a more long-lived object gets allocated during th…
Python is not efficient, but the bug you described seems like a problem with that implementation of malloc...
I'd say it's a problem in the interaction of that specific version of python with that specific malloc. From a performance perspective we could not afford to simply force malloc to only use mmap for every small allocation (as it would thrash the TLB and/or balloon memory usage).
Something they could have done inside Python is manually implement a short-lived objects space that lives entirely in pre-allocated memory, preventing the repeated allocations.
I should have mentioned that the app in particular was running python2.7 - some preliminary tests on porting to python3 seemed to indicate that the issue was significantly less severe there, though the problem only occurred very infrequently that restarting the affected processes when they started exceeding a memory threshold was a more cost-effective solution that spending ages debugging the interactions.
Re: Fixing a Tough Memory Leak in Python
#6When I worked on a Python app that did real-time processing of (very large) lists of python dicts, merely processing them in a list comprehension caused "leaks" causing unbound virtual memory growth. The issue was that that for small allocations, eg. interned strings, malloc would use the "brk" call rather than "mmap". "brk" simply bumps the top of heap pointer, so if a more long-lived object gets allocated during th…
Can you construct a simple example case of that? I think you should report that upstream.
It's in the interaction between python and malloc and also migrating to python3 made the issue mostly go away, which was sufficient for our purposes.
Re: Fixing a Tough Memory Leak in Python
#7The title seems to suggest that there's a memory leak in (C)Python, and while it's tedious to debug mixed Python+C code it's a Nice
Numba memory leak rather than one in Python or numpy.
Re: Fixing a Tough Memory Leak in Python
#8Earlier quoted context omitted.
Python is not efficient, but the bug you described seems like a problem with that implementation of malloc...
I'd say it's a problem in the interaction of that specific version of python with that specific malloc. From a performance perspective we could not afford to simply force malloc to only use mmap for every small allocation (as it would thrash the TLB and/or balloon memory usage). Something they could have done inside Python is manually implement a short-lived objects space that lives entirely in pre-allocated memory,…
> I'd say it's a problem in the interaction of that specific version of python with that specific malloc. From a performance perspective we could not afford to simply force malloc to only use mmap for every small allocation (as it would thrash the TLB and/or balloon memory usage).
And other option might have been to switch to a malloc using mmap'ed size-class arenas like jemalloc or somesuch?
Re: Fixing a Tough Memory Leak in Python
#9When I worked on a Python app that did real-time processing of (very large) lists of python dicts, merely processing them in a list comprehension caused "leaks" causing unbound virtual memory growth. The issue was that that for small allocations, eg. interned strings, malloc would use the "brk" call rather than "mmap". "brk" simply bumps the top of heap pointer, so if a more long-lived object gets allocated during th…
If a generator expression was a valid option, did it change what happened with your leak?
Re: Fixing a Tough Memory Leak in Python
#10When I worked on a Python app that did real-time processing of (very large) lists of python dicts, merely processing them in a list comprehension caused "leaks" causing unbound virtual memory growth. The issue was that that for small allocations, eg. interned strings, malloc would use the "brk" call rather than "mmap". "brk" simply bumps the top of heap pointer, so if a more long-lived object gets allocated during th…
Python uses its own allocator for small sizes (below 512 bytes if I remember correctly). I've had a similar problem to yours. See https://glandium.org/blog/?p=3698 and the followup
https://glandium.org/blog/?p=3723 . I have a glibc bug open about the issue https://sourceware.org/bugzilla/show_bug.cgi?id=23416 .