Live data from Hacker News

Next Generation Out of Band Garbage Collection

railsatscale.com

61–70 of 87 posts

Re: Next Generation Out of Band Garbage Collection

#61
post #47

Earlier quoted context omitted.

A similar strategy is to serialize the object and store it in-process but off heap. This is useful when the values are private to the process, and/or they don't need to survive a crash, and/or you need to avoid the network overhead. Access times are often 100x-1000x faster.

This is something I've been thinking about for a while. What if we create a language where all object references are explicitly specified to be either request-scoped or application-scoped. Don't allow application-scoped objects reference request-scoped objects. Allow to manually upgrade a reference from request-scoped to application-scoped if needed. That would allow us to have ephemeral per-request heaps which are t…

Given that Rust seems to be the generalized solution to this problem, would a viable prototype just be a Rust HTTP server with a Ruby interpreter embedded in it? Write the code in some kind of Ruby DSL which then feeds back into Rust?

I ask because I have embedded Ruby in applications before, and I'm looking for an excuse to do it in Rust.

Re: Next Generation Out of Band Garbage Collection

#62
post #44
post #42

> Ideally in a web application, aside from some in-memory caches, no object allocated as part of a request should survive longer than the request itself. Any object that does is probably something that should be eagerly loaded during boot, or some state that is leaking between requests. As such, any object promoted to the old generation during a request cycle is very unlikely to be immortal, so promoting it is wastef…

That's pretty much what I'm hinting at at the end when I mention minor GC. I don't think doing it after each request would be sensible, but counter intuitively, the time it takes to run GC isn't proportional to amount of garbage to collect, but to the number of live objects left (ignoring some minor things like finalizers). So on paper at least we could run a minor GC for very cheap after each request, but there's li…

Would it be possible for the allocator/GC to know what allocations are made within a request and make a generation for specifically for it? Allocations too big to fit would be made like usual

Re: Next Generation Out of Band Garbage Collection

#63
post #47

Earlier quoted context omitted.

A similar strategy is to serialize the object and store it in-process but off heap. This is useful when the values are private to the process, and/or they don't need to survive a crash, and/or you need to avoid the network overhead. Access times are often 100x-1000x faster.

This is something I've been thinking about for a while. What if we create a language where all object references are explicitly specified to be either request-scoped or application-scoped. Don't allow application-scoped objects reference request-scoped objects. Allow to manually upgrade a reference from request-scoped to application-scoped if needed. That would allow us to have ephemeral per-request heaps which are t…

"What if we create a language where all object references are explicitly specified to be either request-scoped or application-scoped."

I've done this in both C and C++.

The downside of automatic memory management is you have to accept the decisions the memory manager makes.

Still, generational GC like in Ruby and Python essentially attempts to discern the lifetime of allocations, and it gets it right most of the time.

Re: Next Generation Out of Band Garbage Collection

#64
post #62
post #44

Earlier quoted context omitted.

That's pretty much what I'm hinting at at the end when I mention minor GC. I don't think doing it after each request would be sensible, but counter intuitively, the time it takes to run GC isn't proportional to amount of garbage to collect, but to the number of live objects left (ignoring some minor things like finalizers). So on paper at least we could run a minor GC for very cheap after each request, but there's li…

Would it be possible for the allocator/GC to know what allocations are made within a request and make a generation for specifically for it? Allocations too big to fit would be made like usual

That's already what we effectively have.

Since objects cannot be promoted to the old generation inside the request cycle, objects in the new gen are request allocated objects.

So if we were to eagerly trigger a minor GC after a request, we'd have very little objects to scan, and only need to sweep garbage, which is only a small fraction of time spent in GC.

Re: Next Generation Out of Band Garbage Collection

#65
post #29

All the other virtual machines that support GC need to look at the JVM's ZGC and Shenandoah. Sub-millisecond pause times with terabyte heaps.

I think we should be careful when correlating heap size with how long the collection should take. Also, I really want ZGC in .NET runtime, but I don't think I'll ever get support for it first party. There's some kind of principled ideologue holdout situation going on over at Microsoft. Every time I get into it with one of their engineers I'm sent to some impotent "please may I have a temporary GC exemption" API. All…

ZGC does not stand for zero. It stands for Z Garbage Collector. It's a next-generation GC implementation for OpenJDK that focuses on low pause time while supporting very large heap sizes. It does not "not collect garbage".

You could try using https://github.com/kkokosa/UpsilonGC and seeing if it still works.

At the end of the day for anything performance-related you can just write code with manual memory management with RAII patterns via IDisposable on structs and get code that performs closely to C++ or Rust. It's also necessary to understand if this is a good idea at all - most of the time you do want to just rely on GC.

Re: Next Generation Out of Band Garbage Collection

#66
post #29

Earlier quoted context omitted.

I think we should be careful when correlating heap size with how long the collection should take. Also, I really want ZGC in .NET runtime, but I don't think I'll ever get support for it first party. There's some kind of principled ideologue holdout situation going on over at Microsoft. Every time I get into it with one of their engineers I'm sent to some impotent "please may I have a temporary GC exemption" API. All…

ZGC does not stand for zero. It stands for Z Garbage Collector. It's a next-generation GC implementation for OpenJDK that focuses on low pause time while supporting very large heap sizes. It does not "not collect garbage". You could try using https://github.com/kkokosa/UpsilonGC and seeing if it still works. At the end of the day for anything performance-related you can just write code with manual memory management w…

> ZGC does not stand for zero. It stands for Z Garbage Collector.

Apologies - I was attempting to referring to "absolutely no" garbage collection path. I was thinking of Epsilon [0].

> It's also necessary to understand if this is a good idea at all - most of the time you do want to just rely on GC.

Assume we are building a cruise missile flight computer. I have enough ram for ~100 hours of flight if we never clean up any allocations. I only have enough fuel for 8 hours of flight on a good day. Why do I still need a garbage collector? All I need is a garbage generator. The terminal ballistics and warhead are the "out of band" aspects in this arrangement.

> You could try using https://github.com/kkokosa/UpsilonGC and seeing if it still works.

I've spent weeks on this exact thing. I cannot get it to work. This gets me back to the first party support aspect.

[0] https://openjdk.org/jeps/318

Re: Next Generation Out of Band Garbage Collection

#67
post #57

Earlier quoted context omitted.

I don't know if you're joking or not but that is exactly true. Meta went as far as creating their own PHP engine and then a new PHO compatible language because they didn't have the resources to switch from PHP. Instagram is presumably in the same position. Switching language is basically impossible once you have a certain amount of code. I'm sure they were aware of the performance issues with Python but they probably…

Well, Facebook also created their own hacked up version of PHP (called Hack) that's presumably easier to migrate PHP to. Hack is actually surprisingly pleasant, basically about the best language they could have made starting from PHP. (I know, that's damning with faint praise. But I actually mean this unironically. It has TypeScript vibes.)

Yes I mentioned that.

Re: Next Generation Out of Band Garbage Collection

#68
post #58

Earlier quoted context omitted.

Cinder's benchmarks don't seem "like Java" performance, given they aren't that far off cython. https://github.com/facebookincubator/cinder/blob/cinder/3.8/...

CPython itself has seen lots of performance improvements recently. Benchmarks on CPython 3.12 take about half the time they took on CPython 3.9.

Yeah it's definitely welcome, but even if it is double the performance (doesn't seem to be quite there in my experience) fast languages are still 25-50x faster. It's like walking twice as fast when the alternative is driving.

Re: Next Generation Out of Band Garbage Collection

#69
post #56

Earlier quoted context omitted.

> The joke is that if Meta thought that replacing all the Python code they have with something else was worth it, they'd have done it already. "Worth it" depends on both how much performance improvement you get, and how hard it is to replace. Did you consider maybe the rewriting effort is so humongous that it is not worth doing despite large performance improvements? Thus making the joke not funny at all...

That's exactly the joke though. Every time Ruby (or Python) is discussed on HN we get the same old tired question of "why don't they just rewrite in Rust". But that's some silly engineer tunnel vision, squeezing the very last bit of performance out of a system isn't a goal in itself. You just need it to be efficient enough that it cost you significantly less to run that the amount of revenue it brings you. I can bet…

It's usually dismissed because companies think short term, and switching languages is a project with huge short term disadvantages and huge long term advantages.

Re: Next Generation Out of Band Garbage Collection

#70
> no object allocated as part of a request should survive longer than the request itself

So I've spent a lot of time doing Hack (and PHP) as well as Java, Python and other languages. For me, as far as serving HTTP requests goes, Hack/PHP are almost the perfect language. Why?

1. A stateless functional core. There's no loading of large libraries, which is an issue with Python and Java in certain paradigms. The core API us just functions that mean startup costs for a non-stateful service are near zero;

2. The model, as alluded to the above quote, basically creates temporary objects and then tears everything down at the end of the request. It's so much more difficult to leak resources this way as opposed to, say, a stateful Java or C++ server. PHP got a lot of hate unjustly for its "global" scope when in fact it's not global at all. "Global" in PHP/Hack is simply request-scoped and pretty much every language offers request-scoping;

3. There's no threading. Hack, in particular, uses a cooperative async/await model. Where you'd normally create threads (eg making a network request), that's handled by the runtime to make an async/await call out of non-blocking I/O. You never have to deal with mutexes, thread starvation, thread pools, lock ups, etc. You never want to deal with that in "application" or "product" code. Never.

So this article is specific to Ruby-on-Rails, which obviously still has persistent objects, hence the need for GC still.

How Facebook deals with this is kinda interesting. Most FB product code uses an in-memory write-through graph database (called TAO, backed to MySQL). There is an entity model in Hack on top of this that does a whole bunch of stuff like enforcing privacy (ie you basically never talk to TAO directly and if you do, you're going to have to explain why that's necessary, and you absolutely never talk to MySQL directly).

But the point is that persistent entities are request-scoped as well (unlike RoR I guess?).

Post reply on HN