41–50 of 87 posts
All the other virtual machines that support GC need to look at the JVM's ZGC and Shenandoah. Sub-millisecond pause times with terabyte heaps.
And yes, we're aware of ZGC &co https://www.eightbitraptor.com/presentations/RubyKaigi2023-m...
So could each request clean up it's own garbage when it finishes, so then they should never need any global garbage collection?
Earlier quoted context omitted.
I bet the engineers at Instagram were unaware of pythons performance profile when they chose it, you should let them know that they should just switch to a different language.
Meta is just a small startup though, they probably don't have enough resources nor the skills to switch to a better language even after they've heard the gospel.
> Ideally in a web application, aside from some in-memory caches, no object allocated as part of a request should survive longer than the request itself. Any object that does is probably something that should be eagerly loaded during boot, or some state that is leaking between requests. As such, any object promoted to the old generation during a request cycle is very unlikely to be immortal, so promoting it is wastef…
I don't think doing it after each request would be sensible, but counter intuitively, the time it takes to run GC isn't proportional to amount of garbage to collect, but to the number of live objects left (ignoring some minor things like finalizers).
So on paper at least we could run a minor GC for very cheap after each request, but there's likely some better heuristics given currently the median request already spent less than 1ms in GC, so after every requests might be overdoing it.
Also even if we were doing that, many requests would still have to run GC because they allocate more than there is memory available, so they need to clean their own garbage to continue, you can't delay GC indefinitely.
But at least now, endpoints that spend too much time in GC are responsible for their own demise, so engineers responsible for a given endpoint performance have a clear signal that they should allocate less, whereas before it could easily discounted as being caused by lots of garbage left over by another collocated endpoint.
Earlier quoted context omitted.
Meta is just a small startup though, they probably don't have enough resources nor the skills to switch to a better language even after they've heard the gospel.
The joke is that Facebook literally did, right?
Last time I checked they're working on improving Python performance instead (yes I know they forked it into Cinder, but they're trying to upstream their optimizations [0]). Which is very similar to what we're doing at Shopify.
Of course 100% of Instagram isn't in Python, I'm certain there's lots of supporting services in C++ etc, but AFAIK the Instagram "frontend" is still largely a Python/Django app.
The joke is that if Meta thought that replacing all the Python code they have with something else was worth it, they'd have done it already.
> Ideally in a web application, aside from some in-memory caches, no object allocated as part of a request should survive longer than the request itself. This is one of those areas where out of process caching wins. In process caching has a nasty habit of putting freshly created objects into collections that have survived for days or hours, creating writes in the old generation and back references from old to new. Go…
Earlier quoted context omitted.
Author here. Agreed. We have some facility for out of process caching (node local memcached), and I frequently have to argue with colleagues that it's generally preferable to in-process caching.
A similar strategy is to serialize the object and store it in-process but off heap. This is useful when the values are private to the process, and/or they don't need to survive a crash, and/or you need to avoid the network overhead. Access times are often 100x-1000x faster.
That would allow us to have ephemeral per-request heaps which are torn down after every request at once. In-request garbage collections are super-fast. Application-scoped objects are never collected (i.e. no major collections).
Wouldn't this simple model solve most problems? Basically, a very simple equivalent to Rust's lifetimes tailored to web services without all the complexity, and much less GC overhead than in traditional GC systems.
All the other virtual machines that support GC need to look at the JVM's ZGC and Shenandoah. Sub-millisecond pause times with terabyte heaps.
C4 still smokes them both, doesn't it?
Earlier quoted context omitted.
A similar strategy is to serialize the object and store it in-process but off heap. This is useful when the values are private to the process, and/or they don't need to survive a crash, and/or you need to avoid the network overhead. Access times are often 100x-1000x faster.
This is something I've been thinking about for a while. What if we create a language where all object references are explicitly specified to be either request-scoped or application-scoped. Don't allow application-scoped objects reference request-scoped objects. Allow to manually upgrade a reference from request-scoped to application-scoped if needed. That would allow us to have ephemeral per-request heaps which are t…
Earlier quoted context omitted.
I bet the engineers at Instagram were unaware of pythons performance profile when they chose it, you should let them know that they should just switch to a different language.
Meta is just a small startup though, they probably don't have enough resources nor the skills to switch to a better language even after they've heard the gospel.
Instagram is presumably in the same position. Switching language is basically impossible once you have a certain amount of code. I'm sure they were aware of the performance issues with Python but they probably said "we'll worry about it later" when they were a small startup and now it's too late.