How I spent two weeks hunting a memory leak in Ruby (2015) • Hacker News Board

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#21

Post by nateberkopec » Tue, Sep 13, 2016, 10:22 PM UTC

samsk wrote: post #20

shameless plug: I've made malloc hooking preload library for hunting such bugs, that automatizes most of the described work - https://github.com/samsk/log-malloc2

This looks really, really cool. You should download the bugged version of this gem and make a screencast showing how your tool makes this process easier.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#22

Post by jmcgough » Tue, Sep 13, 2016, 10:42 PM UTC

darkhorn wrote: post #18

Does PHP have similar issues? I think it shouldn't.

PHP has garbage collection now, so it's a lot harder to cause a memory leak. Before PHP 5.3.0, cyclical references were one of the biggest causes of memory leaks.

However, you can still create memory leaks just like you could in Ruby - dynamic allocation in a c extension that isn't freed up (like in this story), or creating objects that can't be garbage collected (like a slowing expanding array in a global variable).

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#23

Post by lotyrin » Tue, Sep 13, 2016, 10:48 PM UTC

darkhorn wrote: post #18

Does PHP have similar issues? I think it shouldn't.

PHP extensions in C are pretty rare since it's so hard to distribute them (outside a few exceptions that exist in common package management systems), so in that sense, I guess?

But no, a PHP extension in C can have a memory leak like any other C code. It's not at all uncommon actually, since there are fewer consequences; CGI mode PHP cleans up after every request, FastCGI mode PHP by default restarts workers after a number of requests because of the class of effects (unwanted state) that includes memory leaks.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#24

Post by [deleted] » Tue, Sep 13, 2016, 11:28 PM UTC

[deleted]

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#25

Post by kyledrake » Tue, Sep 13, 2016, 11:34 PM UTC

Interesting post, but I did want to chime in quickly and say that it's pretty absurd to only have 1GB for a web application in 2016, even for a small one. This is why I've been dismissive of using Heroku for my projects, even though I run a lean stack. They have higher RAM options, but they are incredibly expensive. For the cost of a 14GB Heroku Dyno, I can buy a dedicated server off ebay with 32GB ECC every single month. I don't use all of that RAM, but it's nice to have some extra RAM to toss around when you need to in a pinch.

I get that it means you have to "run a server", and insert arguments for expensive cloud providers vs DIY servers here but I don't think it's any less crazy than being forced to chase a GC white whale for two weeks on a tiny memory leak to avoid a huge rate hike on your hosting bill.

On an aside, I've only had one problematic memory leak with ruby ever (the infamously leaky RMagick), I threw this in every time I used the lib and it solved it for me:

  GC.start(full_mark: true, immediate_sweep: true)

Bit of a performance hit, but not enough to cause a problem for my use case.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#26

Post by simcop2387 » Tue, Sep 13, 2016, 11:36 PM UTC

busterarm wrote: post #16

bru wrote: post #8

About the patch at https://github.com/vmg/redcarpet/pull/516/files , couldn't the update simply have been the following? - return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, NULL, rndr); + return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, xfree, rndr);

C is not really my area of expertise, but doesn't xfree have alternatives? like free, freeif, etc? I'd much rather have to change that later in one place than several (if there end up being more calls to xfree in the code) if at some point the implementation needs to change again.

So a from a quick googling it looks like xfree()[1] is a ruby specific thing, and should be used internally for everything allocated by ruby. It also looks like it's got the same signature as the function created for this so that should work there too. [2]

[1] http://clalance.blogspot.com/2011/01/writing-ruby-extensions... [2] http://inferior-products.com/docs/userdocs/ruby19/html/d8/d1...

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#27

Post by busterarm » Tue, Sep 13, 2016, 11:59 PM UTC

simcop2387 wrote: post #26

busterarm wrote: post #16

Earlier quoted context omitted.

C is not really my area of expertise, but doesn't xfree have alternatives? like free, freeif, etc? I'd much rather have to change that later in one place than several (if there end up being more calls to xfree in the code) if at some point the implementation needs to change again.

So a from a quick googling it looks like xfree()[1] is a ruby specific thing, and should be used internally for everything allocated by ruby. It also looks like it's got the same signature as the function created for this so that should work there too. [2] [1] http://clalance.blogspot.com/2011/01/writing-ruby-extensions... [2] http://inferior-products.com/docs/userdocs/ruby19/html/d8/d1...

...Are you sure? I was pretty sure that it gets it from the same place as xmalloc...wouldn't that be in libc?

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#28

Post by mikekchar » Wed, Sep 14, 2016, 12:01 AM UTC

kyledrake wrote: post #25

Interesting post, but I did want to chime in quickly and say that it's pretty absurd to only have 1GB for a web application in 2016, even for a small one. This is why I've been dismissive of using Heroku for my projects, even though I run a lean stack. They have higher RAM options, but they are incredibly expensive. For the cost of a 14GB Heroku Dyno, I can buy a dedicated server off ebay with 32GB ECC every single m…

> it's pretty absurd to only have 1GB for a web application in 2016

While I don't for a minute doubt what you say, as an old guy, the absurdity of 1GB being insufficient for serving up web pages hits me pretty hard.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#29

Post by mrud » Wed, Sep 14, 2016, 12:04 AM UTC

busterarm wrote: post #27

simcop2387 wrote: post #26

Earlier quoted context omitted.

So a from a quick googling it looks like xfree()[1] is a ruby specific thing, and should be used internally for everything allocated by ruby. It also looks like it's got the same signature as the function created for this so that should work there too. [2] [1] http://clalance.blogspot.com/2011/01/writing-ruby-extensions... [2] http://inferior-products.com/docs/userdocs/ruby19/html/d8/d1...

...Are you sure? I was pretty sure that it gets it from the same place as xmalloc...wouldn't that be in libc?

There is no xmalloc in the libc.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#30

Post by busterarm » Wed, Sep 14, 2016, 12:06 AM UTC

mrud wrote: post #29

busterarm wrote: post #27

Earlier quoted context omitted.

...Are you sure? I was pretty sure that it gets it from the same place as xmalloc...wouldn't that be in libc?

There is no xmalloc in the libc.

Okay, interesting. I guess I've seen it before, so I didn't think it was Ruby-specific, but I also haven't had to do that kind of programming in a very super long time.

Publib then? I'm finding this when I search: http://man.cx/xfree(3) http://man.cx/publib(3)

Also, are you sure? I see xmalloc in glibc all over the internet...