Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory leak proof every C program

flak.tedunangst.com

Post reply on HN
Print view View original

111–120 of 175 posts

Re: Memory leak proof every C program

#111

Post by akoboldfrying »

bluish29 wrote: post #101
mysterydip wrote: post #97

Earlier quoted context omitted.

While I'm sure it would bring its own problems, could you just stick a semaphore or similar around the function?

Yes, but it will not work in a generic case because you have to handle the situations of deadlocks (i.e recursive mutexes for nested locking ..etc). Also you will introduce more problems of creating performance deadlocks. There will not be out of box solutions to all these. Once you start considering solving case on its own then this does not provide out of box solution as the original premise.

I think the only way this design could deadlock is if a signal handler calls malloc() -- specifically, if the interrupted code on that CPU thread was partway through a malloc() call of its own, having locked the mutex but not yet unlocked it.

In any other scenario, if some execution thread reaches the point where it returns from calling lock() on the mutex guarding malloc(), it must eventually reach the call to unlock().

Re: Memory leak proof every C program

#112

Post by tom_ »

bluGill wrote: post #90
kbenson wrote: post #54

Earlier quoted context omitted.

The difference is that if you're writing a program you know the scope of use of all libraries, whether they be externally loaded or internal abstraction boundaries, and also know the scope of use of the program, and can make a call as to whether cleanup during opetation is required.

Not really. In theory you can, but in practice there are parts written by a different team and you don't know the scope of there parts. I also know someone who maintains some Clinton era encryption. that code is controlled who can know about it as obsecurity was all you were allowed. There are other pathalogical cases where you can't know everything about your program-

But those parts, if they exist, are irrelevant to the current discussion, because they are not the parts you are writing.

Re: Memory leak proof every C program

#113

Post by [deleted] »

lgg wrote: post #82

There is a bug here... Clearly the author intended to cache the value of nextmalloc to avoid calling dlsym() on every malloc. The correct code should be: static void *(*nextmalloc)(size_t) = NULL; if (!nextmalloc) nextmalloc = dlsym(RTLD_NEXT, "malloc"); } Somehow the fact that the optimization is incorrectly missed here feels appropriate ;-)

[deleted]

Re: Memory leak proof every C program

#114

Post by JB_Dev »

rco8786 wrote: post #41
enriquto wrote: post #3

This is the core idea: > It is [...] entirely optional to call free. If you don’t call free, memory usage will increase over time, but technically, it’s not a leak. As an optimization, you may choose to call free to reduce memory, but again, strictly optional. This is beautiful! Unless your program is long-running, there's no point of ever calling free in your C programs. The system will free the memory for you when…

Reminds me of the HFT shop that built in Java and simply turned the garbage collector off. Then when the market closed they would restart the process for the next day.

I’ve done something similar in one of our production services. There was a problem with extremely long GC pauses during Gen2 garbage collection (.NET uses a multigenerational GC design). Pauses could be many seconds long or more than a minute in extreme cases.

We found the underlying issue that caused memory pressure in the Gen2 region but the fix was to change some very fundamental aspects of the service and would need to have some significant refactoring. Since this was a legacy service (.net framework) that we were refactoring anyway to run in new .NET (5+), we decided to ignore the issue.

Instead we adjusted the GC to just never do the expensive Gen2 collections (GCLatencyMode) and moved the service to run on higher memory VMs. It would hit OOM every 3 days or so, so we just set instances to auto-restart once a day.

Then 1 year later we deployed the replacement for the legacy service and the problem was solved.

Re: Memory leak proof every C program

#115

Post by lifthrasiir »

lpapez wrote: post #10

I bet this is what some people on my team would come up with if the ticket acceptance criteria said "program must not leak memory when checked with valgrind"

You can easily turn non-leaking program into a faster and leaking program, but the inverse direction is hard, so that criterion is entirely justified. Any optimization of this sort should be guarded against a compile time switch that simply swaps `free` with a placeholder. I think CPython did this for a long time, before the global initialization step was completely removed.

Re: Memory leak proof every C program

#116

Post by lifthrasiir »

tverbeure wrote: post #92

In the same spirit, here's a fun talk about tool that guarantees 100% line coverage for your Python tool: https://www.youtube.com/watch?v=A-cjMRsHcXI After running your test suite, it kicks into action and deletes all the lines of your code that were never executed.

It is actually a viable strategy if you are heavily optimizing for the executable size [1]. It's essentially a radical dead code elimination scheme.

[1] https://fgiesen.wordpress.com/2012/04/08/metaprogramming-for...

Re: Memory leak proof every C program

#118

Post by queuebert »

HideousKojima wrote: post #51
rco8786 wrote: post #41

Earlier quoted context omitted.

Reminds me of the HFT shop that built in Java and simply turned the garbage collector off. Then when the market closed they would restart the process for the next day.

Or the missile firmware where the missile is going to explode before they run out of memory: https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...

God, that email takes me back to days when email was much higher signal-to-noise and sig lines were for color, not credentials.

Re: Memory leak proof every C program

#119

Post by tarnith »

shortrounddev2 wrote: post #84
tombert wrote: post #80

Earlier quoted context omitted.

I had a friend who worked for one of the big Market Makers, and he told me that they would indeed turn the GC off, but what they'd do is just pre-allocate everything into bigass arrays before-hand, and have incrementers to simulate the "new" keyword. They might do this in something more or less like a threadlocal to avoid having to deal with locks or race conditions or anything like that.

A lot of game engines do this, avoiding the cost of a malloc lookup by just allocating everything as a stack

Seen some audio engines that do this too. You can come up with an elegant non-blocking lock free allocator and pass updates from the user thread out to system and reserve the memory, or you can just up front allocate everything that's needed.

If it's a fixed function synth, sometimes just allocating everything up front makes more sense.

Post reply on HN