Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I spent two weeks hunting a memory leak in Ruby (2015)

be9.io

Post reply on HN
Print view View original

41–50 of 71 posts

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#41

Post by recurser »

kyledrake wrote: post #25

Interesting post, but I did want to chime in quickly and say that it's pretty absurd to only have 1GB for a web application in 2016, even for a small one. This is why I've been dismissive of using Heroku for my projects, even though I run a lean stack. They have higher RAM options, but they are incredibly expensive. For the cost of a 14GB Heroku Dyno, I can buy a dedicated server off ebay with 32GB ECC every single m…

I work on the project discussed in the article. We're running puma in clustered mode, with multiple workers per dyno and multiple threads per worker. 1GB is plenty in this context - each worker is pretty lightweight. At the time this was written we were running ~10 2x dynos, but we've since switched to 3 performance-m dynos with more puma threads/workers.

> I don't think it's any less crazy than being forced to chase a GC white whale for two weeks on a tiny memory leak to avoid a huge rate hike on your hosting bill.

The main issue we were facing wasn't the hosting bill, but high (for us) traffic, that was leaking memory on every request. Under low traffic it was unnoticeable, but at peak load the leak would cause dyno memory to max out pretty quickly, which would cause timeouts and increase the traffic to other dynos, causing cascading failures. Having more memory available would definitely have made things a lot easier, but we would have run out eventually either way.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#42

Post by always_good »

kyledrake wrote: post #25

Interesting post, but I did want to chime in quickly and say that it's pretty absurd to only have 1GB for a web application in 2016, even for a small one. This is why I've been dismissive of using Heroku for my projects, even though I run a lean stack. They have higher RAM options, but they are incredibly expensive. For the cost of a 14GB Heroku Dyno, I can buy a dedicated server off ebay with 32GB ECC every single m…

    > I did want to chime in quickly and say that
    > it's pretty absurd to only have 1GB for a
    > web application in 2016
For Rails and Java apps, I can see that.

But if the 512MB dyno isn't enough for any of the stacks I use, including Node, then I'm doing something very wrong (or weird).

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#44

Post by asveikau »

bru wrote: post #8

About the patch at https://github.com/vmg/redcarpet/pull/516/files , couldn't the update simply have been the following? - return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, NULL, rndr); + return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, xfree, rndr);

The shortest code you can get away with is not always the best. If the parameter is a structure (which I think it is here, behind the void pointer), it's a good idea to give it its own free function in case later you add additional struct members that will need freeing/cleanup in the same place. (Bonus points if it appears in roughly the same place this thing is malloc'd)

I would have been even more explicit and added a cast to the expected structure pointer type. Entirely useless, except to the human reader.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#45

Post by userbinator »

snissn wrote: post #37
mikekchar wrote: post #28

Earlier quoted context omitted.

> it's pretty absurd to only have 1GB for a web application in 2016 While I don't for a minute doubt what you say, as an old guy, the absurdity of 1GB being insufficient for serving up web pages hits me pretty hard.

Interpreted languages can have pretty high memory overhead. Let's say we have a page in python or php or ruby or whatever, and it needs to fetch 1,000 rows from a database (100 things here, 20 things there, whatever). Each row in the database can be turned into an object several times, one from the model, one from the controller, one for the template/view. On top of that, each object is a composite key/value object o…

Needing to fetch 1k rows from a database is different from needing to keep all that data in memory, however. Several years ago I had to return 1M rows from a database to the client with a PHP script, and it needed less than the 16MB (yes, sixteen megabytes) that the environment of the time was configured with as a memory limit. 1G rows would not have consumed any more memory either, because at any one time the script never needed to keep more than around a dozen cells in memory, each not more than a few KB. Read a cell from the database, process it, output the result and move onto the next one. Essentially O(1) space regardless of how much data there is.

Or are you saying that web applications are now written to retain all that data in memory despite the fact that it's not actually needed, and such stupid algorithms are actually considered acceptable? I think that's pretty ridiculous. The whole idea of database concepts like cursors is that you shouldn't be required to hold the entire resultset in memory, because it might not fit.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#46

Post by foobarian »

kyledrake wrote: post #25

Interesting post, but I did want to chime in quickly and say that it's pretty absurd to only have 1GB for a web application in 2016, even for a small one. This is why I've been dismissive of using Heroku for my projects, even though I run a lean stack. They have higher RAM options, but they are incredibly expensive. For the cost of a 14GB Heroku Dyno, I can buy a dedicated server off ebay with 32GB ECC every single m…

My favorite Ruby memory leak was in a text processing app that was extracting via regex short strings (< 50 chars) out of 10MB+ text files. The short strings were apparently substring-like objects that held on the original 10MB text and indexed into it at a particular offset. Solution? Return the short_string + "" and then the GC did its thing.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#47

Post by userbinator »

asveikau wrote: post #44
bru wrote: post #8

About the patch at https://github.com/vmg/redcarpet/pull/516/files , couldn't the update simply have been the following? - return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, NULL, rndr); + return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, xfree, rndr);

The shortest code you can get away with is not always the best. If the parameter is a structure (which I think it is here, behind the void pointer), it's a good idea to give it its own free function in case later you add additional struct members that will need freeing/cleanup in the same place. (Bonus points if it appears in roughly the same place this thing is malloc'd) I would have been even more explicit and adde…

That sounds like premature generalisation to me.

I'd say use a separate free function when it needs one, which is not the case yet (and might never be.)

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#48

Post by wahern »

userbinator wrote: post #45
snissn wrote: post #37

Earlier quoted context omitted.

Interpreted languages can have pretty high memory overhead. Let's say we have a page in python or php or ruby or whatever, and it needs to fetch 1,000 rows from a database (100 things here, 20 things there, whatever). Each row in the database can be turned into an object several times, one from the model, one from the controller, one for the template/view. On top of that, each object is a composite key/value object o…

Needing to fetch 1k rows from a database is different from needing to keep all that data in memory, however. Several years ago I had to return 1M rows from a database to the client with a PHP script, and it needed less than the 16MB (yes, sixteen megabytes ) that the environment of the time was configured with as a memory limit. 1G rows would not have consumed any more memory either, because at any one time the scrip…

This should be even easier these days. Async I/O platforms like Node.js mean you don't have to worry about a process sticking around while it trickles data to a client. And web frameworks make it easier to deliver datasets asynchronously as JSON to the browser. Generally, there's less excuse to buffer everything server-side, e.g. to generate a full page.

I think the reliance on large frameworks and external libraries is the problem here because they don't generally compose in a way that permits optimizing data flow. That and the usual reduction in average skill required to work in this area.[1] People will spend an eternity arguing over which is the faster hash table algorithm, but have no eye or concern for optimizing the large details even though the relative improvements in performance and scale could be orders of magnitude better and conceptually much easier to implement.

This is why higher-level languages need stackful coroutines, not just promises or async/await. Stackful coroutines make it much easier to implement and compose space-efficient iterators, for example, in the context of uncooperative libraries, and to do so performantly. People are so obsessed with async I/O they've forgotten why it's so darned useful and how you want to best leverage it. They keep going down the rabbit hole of callbacks and actors, but the fact of the matter is that those things are too complex to use pervasively in the pipeline of individual requests. They're the modern equivalent of gotos--easy to use and abuse, but not what you want to use as an abstraction for tying together different pieces of code because they have poor locality of context, require leaking too much detail at the interface boundaries, are more difficult to refactor, and generally require a high cognitive burden. We want to be lowering cognitive load and other constraints!

[1] To be fair, I was one of those less-skilled newcomers back in the late 90s, abusing the newer technologies and frustrating the grey beards.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#49

Post by asveikau »

userbinator wrote: post #47
asveikau wrote: post #44

Earlier quoted context omitted.

The shortest code you can get away with is not always the best. If the parameter is a structure (which I think it is here, behind the void pointer), it's a good idea to give it its own free function in case later you add additional struct members that will need freeing/cleanup in the same place. (Bonus points if it appears in roughly the same place this thing is malloc'd) I would have been even more explicit and adde…

That sounds like premature generalisation to me. I'd say use a separate free function when it needs one , which is not the case yet (and might never be.)

Call it an overreaction to a past project in which a predecessor scattered free calls over several uses, then I had to add fields.

I don't mind adding 1 useless function per complex type if it saves me those headaches even a small minority of the time, or for the next maintainer. Opinions may differ, but that's me.

Re: How I spent two weeks hunting a memory leak in Ruby (2015)

#50

Post by mwpmaybe »

busterarm wrote: post #14

What a timely post. I had a similar issue that I was tracing last week that did end up being my Ruby code...and it turns out I was modifying a constant like in the example. What a fun read! I've been learning more about Ruby's GC since 2.1 and this got me looking even deeper -- definitely picked up a couple of new tricks/tools from this. Thank you be9!

Freeze your constants!
Post reply on HN