Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nimrod: A new approach to metaprogramming

nimrod-lang.org

Post reply on HN
Print view View original

31–40 of 58 posts

Re: Nimrod: A new approach to metaprogramming

#31

Post by rbehrends »

rdtsc wrote: post #30
rbehrends wrote: post #26

Earlier quoted context omitted.

I have been using Nimrod for some personal projects and internal tools since about version 0.9.0. It is quite usable -- especially the current Github version -- but as you may expect from a 0.x project, it still has some rough edges. These haven't stopped me from being productive with it, but you should be prepared for the occasional "huh?" situation (the most common problem I've had was that when I experimented with…

Thank you for replying. That was a very good overview. Could you elaborate if possible a bit on a thread private heap. I know Erlang's actor and Dart's isolates allow that (which makes it easy for it to implement a concurrency GC). What is the mechanism for creating private heaps (if there is one)? Or is it just by convention as in "just know that from this one thread I only create objects and no other thread will ac…

When a thread is created, it automatically comes with a new thread-local heap. You can send objects to other threads via channels (which will be deep copied to the other heap). Each thread will allocate objects within its own heap by default.

Note that you can also fall back to GC-free allocation with untraced references (using ptr instead of ref) but will then have to manage that part of the memory yourself (i.e., deallocate untraced objects yourself).

Re: Nimrod: A new approach to metaprogramming

#32

Post by rdtsc »

rbehrends wrote: post #31
rdtsc wrote: post #30

Earlier quoted context omitted.

Thank you for replying. That was a very good overview. Could you elaborate if possible a bit on a thread private heap. I know Erlang's actor and Dart's isolates allow that (which makes it easy for it to implement a concurrency GC). What is the mechanism for creating private heaps (if there is one)? Or is it just by convention as in "just know that from this one thread I only create objects and no other thread will ac…

When a thread is created, it automatically comes with a new thread-local heap. You can send objects to other threads via channels (which will be deep copied to the other heap). Each thread will allocate objects within its own heap by default. Note that you can also fall back to GC-free allocation with untraced references (using ptr instead of ref ) but will then have to manage that part of the memory yourself (i.e.,…

I am sold. I really like it!

Re: Nimrod: A new approach to metaprogramming

#33

Post by rbehrends »

RyanZAG wrote: post #16
pcwalton wrote: post #13

Earlier quoted context omitted.

It's deferred reference counting, so the 1-2ms claim is perfectly reasonable because all it has to scan is the stack (though if they want to break cycles they will have to do something costlier). The downsides are all of the downsides of reference counting, plus the fact that last I looked Nimrod's GC is not thread safe. That last one is a huge downside IMHO. I think DRC is interesting but I'm not much of a fan of it…

Deferred means you eventually have to do the work. Same for reference counting - when you free that last element it will have to actually do the freeing. If you've filled up ram and are allocating objects fast then you can either crash or do some garbage collection and if the tree is large it will have to take more than 1-2ms.

Deferred means you eventually have to do the work.

Not quite. What you save with deferred reference counting is what is generally the biggest cost of reference counting: assigning references to local variables and having local variables go out of scope. Deferred reference counting only requires an RC update if you store a reference in a heap object or in a global variable.

You will, of course, necessarily incur the overhead for allocation and deallocation, which one can make cheaper (bump allocation in generational and copying GC, memory regions) but can't entirely get rid of, especially in non-copying/compacting schemes.

Re: Nimrod: A new approach to metaprogramming

#34

Post by pcwalton »

rbehrends wrote: post #33
RyanZAG wrote: post #16

Earlier quoted context omitted.

Deferred means you eventually have to do the work. Same for reference counting - when you free that last element it will have to actually do the freeing. If you've filled up ram and are allocating objects fast then you can either crash or do some garbage collection and if the tree is large it will have to take more than 1-2ms.

Deferred means you eventually have to do the work. Not quite. What you save with deferred reference counting is what is generally the biggest cost of reference counting: assigning references to local variables and having local variables go out of scope. Deferred reference counting only requires an RC update if you store a reference in a heap object or in a global variable. You will, of course, necessarily incur the o…

But you lose the promptness of deallocation. Furthermore, I think the biggest overhead of reference counting is not the count manipulation (especially if you aren't making it thread safe), it's the inability to handle cycles, which DRC does not address.

Re: Nimrod: A new approach to metaprogramming

#35

Post by pcwalton »

rbehrends wrote: post #26
rdtsc wrote: post #12

A GC with a deadline is what caught my attention, seemingly can specify a max pause time (in ms) and GC wont' take longer than that. That is very good for soft realtime stuff -- games, audio processing. Also, Nimrod has a very nice library. Definitely has enough "batteries" in it to get started. http://nimrod-lang.org/lib.html Just a few impressive ones: http client, server, json parsing, actor support, redis db driv…

I have been using Nimrod for some personal projects and internal tools since about version 0.9.0. It is quite usable -- especially the current Github version -- but as you may expect from a 0.x project, it still has some rough edges. These haven't stopped me from being productive with it, but you should be prepared for the occasional "huh?" situation (the most common problem I've had was that when I experimented with…

Wait, if the RC scheme has a cycle collector, how do you ensure there are no CC pauses for acyclic data? Every CC algorithm I know of uses heuristics to determine when to scan the heap for cycles (usually when an RC drops from >2 to 1) and those can result in pauses proportional to the size of the live set on the heap, even when there are no cycles.

Re: Nimrod: A new approach to metaprogramming

#36

Post by rbehrends »

pcwalton wrote: post #35
rbehrends wrote: post #26

Earlier quoted context omitted.

I have been using Nimrod for some personal projects and internal tools since about version 0.9.0. It is quite usable -- especially the current Github version -- but as you may expect from a 0.x project, it still has some rough edges. These haven't stopped me from being productive with it, but you should be prepared for the occasional "huh?" situation (the most common problem I've had was that when I experimented with…

Wait, if the RC scheme has a cycle collector, how do you ensure there are no CC pauses for acyclic data? Every CC algorithm I know of uses heuristics to determine when to scan the heap for cycles (usually when an RC drops from >2 to 1) and those can result in pauses proportional to the size of the live set on the heap, even when there are no cycles.

I am not sufficiently familiar with the inner details of Nimrod's GC implementation to give you a full answer, but, no, you don't have to scan the entire live heap to reclaim cycles.

The technique is called "trial deletion" and only has to traverse potential cycles. Strictly speaking, a type-agnostic implementation may have to traverse all objects reachable from an object whose reference count was decremented since the previous pass, but in a strongly typed language you can skip that for all objects that can't be part of a cycle. Nimrod at least makes use of that information in asgnRefNoCycle() in system/gc.nim.

Obviously, if everything on the heap is part of one big cycle, then, yes, you'll have to scan the entire heap.

Re: Nimrod: A new approach to metaprogramming

#37

Post by rbehrends »

pcwalton wrote: post #34
rbehrends wrote: post #33

Earlier quoted context omitted.

Deferred means you eventually have to do the work. Not quite. What you save with deferred reference counting is what is generally the biggest cost of reference counting: assigning references to local variables and having local variables go out of scope. Deferred reference counting only requires an RC update if you store a reference in a heap object or in a global variable. You will, of course, necessarily incur the o…

But you lose the promptness of deallocation. Furthermore, I think the biggest overhead of reference counting is not the count manipulation (especially if you aren't making it thread safe), it's the inability to handle cycles, which DRC does not address.

You are correct that you won't get the immediacy of reference counting; that's a problem of all memory schemes that aren't naive reference counting. But no, the biggest constant time overhead for naive reference counting comes from the RC updates for literally any pointer manipulation. Not just the overhead of having increments, decrements, and possibly failed branch predictions for every single pointer assignment, but also what it does to cache locality. Deferred reference counting essentially only needs a write barrier not very different from generational or incremental forms of garbage collection. This is why tuned versions of deferred reference counting come pretty close to other tuned GC schemes, but naive reference counting falls short, even in the absence of cycles.

Cycle detection primarily affects pause time, not overhead.

Re: Nimrod: A new approach to metaprogramming

#39

Post by rcthompson »

wwwtyro wrote: post #4

Cool project. How does it accomplish the realtime GC max pauses of 1-2 ms after compiling to javascript? Wouldn't the js implementation's GC have the final say on GC pauses? Thanks.

Probably by compiling to JS code that avoids the JS garbage collector, the way asm.js does.
Post reply on HN