How I spent two weeks hunting a memory leak in Ruby (2015) • Hacker News Board

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#11

Post by yxhuvud » Tue, Sep 13, 2016, 8:33 PM UTC

rgtk wrote: post #4

There is no reason for using native extensions in scripting languages like Ruby when program load is pretty low (especially when they are used for performance reason, not binding). Great insight, though. Author described this experience as it was a great venture... in hindsight I suppose :-)

The reason for the binding can be that no-one can be assed to reimplement the functionality. It is not as if anyone expect the scripting languages to implement their own versions of openssl.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#12

Post by scott_s » Tue, Sep 13, 2016, 8:41 PM UTC

rgtk wrote: post #4

There is no reason for using native extensions in scripting languages like Ruby when program load is pretty low (especially when they are used for performance reason, not binding). Great insight, though. Author described this experience as it was a great venture... in hindsight I suppose :-)

Your criticism really applies to the gem authors, which to me, does not make sense. If I, as a library developer, want my library to be used on the critical path in heavy-load situations, and the library is for a language like Ruby, I'm going to figure out which part of my library are the performance critical parts, and implement them in a native language.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#13

Post by amalag » Tue, Sep 13, 2016, 8:49 PM UTC

rgtk wrote: post #4

There is no reason for using native extensions in scripting languages like Ruby when program load is pretty low (especially when they are used for performance reason, not binding). Great insight, though. Author described this experience as it was a great venture... in hindsight I suppose :-)

Now that people are figuring out how to do this in Rust things should get a lot better.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#14

Post by busterarm » Tue, Sep 13, 2016, 8:51 PM UTC

What a timely post.

I had a similar issue that I was tracing last week that did end up being my Ruby code...and it turns out I was modifying a constant like in the example.

What a fun read! I've been learning more about Ruby's GC since 2.1 and this got me looking even deeper -- definitely picked up a couple of new tricks/tools from this. Thank you be9!

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#15

Post by wyldfire » Tue, Sep 13, 2016, 8:52 PM UTC

These are always interesting bug hunts.

I tried to add ASan to the travis config for this project but I couldn't quite figure out how to change CFLAGS and/or CC. Never used ruby but interwebs hinted that the bundle config/install commands might accept "--cc" and "--with-cflags" commands. It's ignored when I tried it though [1].

[1] https://travis-ci.org/androm3da/redcarpet/jobs/159711520

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#16

Post by busterarm » Tue, Sep 13, 2016, 8:58 PM UTC

bru wrote: post #8

About the patch at https://github.com/vmg/redcarpet/pull/516/files , couldn't the update simply have been the following? - return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, NULL, rndr); + return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, xfree, rndr);

C is not really my area of expertise, but doesn't xfree have alternatives? like free, freeif, etc? I'd much rather have to change that later in one place than several (if there end up being more calls to xfree in the code) if at some point the implementation needs to change again.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#17

Post by mnarayan01 » Tue, Sep 13, 2016, 8:58 PM UTC

rgtk wrote: post #10

mnarayan01 wrote: post #7

Earlier quoted context omitted.

> when program load is pretty low The leak was in a gem.

Gem is still library albeit its name.

1. AFAIK it's the most popular Markdown gem around (ref. https://www.ruby-toolbox.com/categories/markup_processors) so using it in a project is more "default" than it is "optimization".

2. With over 10M downloads, I assume at least some of them spend large percentages of time using it.

3. It's Markdown; replicating the idiosyncrasies of SmartyPants (https://daringfireball.net/projects/smartypants/) in a different language is not something many people are going to want to do.

I'm not saying that people should or should not use Redcarpet; just that it's unlikely that anyone engaged in "premature optimization" here.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#18

Post by darkhorn » Tue, Sep 13, 2016, 9:09 PM UTC

Does PHP have similar issues? I think it shouldn't.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#19

Post by vemv » Tue, Sep 13, 2016, 9:19 PM UTC

While the described debug process deserves nothing but my applause, I wonder - could the issue have been more easily diagnosed?

Ideally much less tooling would have been needed.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#20

Post by samsk » Tue, Sep 13, 2016, 9:52 PM UTC

shameless plug: I've made malloc hooking preload library for hunting such bugs, that automatizes most of the described work - https://github.com/samsk/log-malloc2