Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Interesting things about the Lua interpreter (2020)

thesephist.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: Interesting things about the Lua interpreter (2020)

#21

Post by Lyngbakr »

VWWHFSfQ wrote: post #20
sireat wrote: post #18

I suppose that's the price for the small core but I do wish Lua had string interpolation. Instead there are like 5 hacky ways to do it: http://lua-users.org/wiki/StringInterpolation

As with nearly everything in Lua, if you want bells-and-whistles then you need to find a library. I've been writing a lot of extensions in Rust using mlua [1] which has been fantastic. I've written so much FFI code for Lua it's ridiculous. Rust mlua has been a total game-changer for me. [1] https://github.com/khvzak/mlua

mlua looks amazing. Is that the go-to library for Lua bindings in Rust or are there still several options jostling for position?

Re: Interesting things about the Lua interpreter (2020)

#22

Post by VWWHFSfQ »

Lyngbakr wrote: post #21
VWWHFSfQ wrote: post #20

Earlier quoted context omitted.

As with nearly everything in Lua, if you want bells-and-whistles then you need to find a library. I've been writing a lot of extensions in Rust using mlua [1] which has been fantastic. I've written so much FFI code for Lua it's ridiculous. Rust mlua has been a total game-changer for me. [1] https://github.com/khvzak/mlua

mlua looks amazing. Is that the go-to library for Lua bindings in Rust or are there still several options jostling for position?

mlua is pretty much the standard for Rust at this point. There are a few others but they're not nearly as comprehensive and they all suffer from various safety issues that require unsafe{}. mlua doesn't require any unsafe usage in its API.

Re: Interesting things about the Lua interpreter (2020)

#23

Post by cardanome »

giraffe_lady wrote: post #7

Timing on this is confusing. Article was written in 2020 about a paper published in 2003 or so about the design of lua 5.0. It's still a really insightful and approachable paper that's worth reading, and it does still help understand the constraints and approach of lua. But the current "old" version of the language is 5.2 released in 2011, and there have been a couple major versions after that as well. So some of it…

It is still relevant. The newer versions after 5.1 just add minor stuff.

Lua is basically done as a language.

Re: Interesting things about the Lua interpreter (2020)

#24

Post by VWWHFSfQ »

babelfishstudio wrote: post #2

I love that they went from a single-pass interpreter to a byte-code virtual machine, and now like PHP, also has direct access to C & system libraries! Lua has come a long way since it's advent.

The article is about Lua 5.0 which is obsolete. The only relevant versions of Lua are 5.1 (LuaJIT/nginx + Roblox/Luau) and 5.3+

Re: Interesting things about the Lua interpreter (2020)

#25

Post by xenadu02 »

ithkuil wrote: post #13
ufo wrote: post #4

> The 5.0 VM is a register machine, which operates on a set of virtual registers that can store and act on the local variables of a function, in addition to the traditional runtime stack. This is a common source of confusion, because the name "register machine" makes people think about CPU registers. However, the registers in a register VM are merely slots in the traditional runtime stack. The difference between a st…

This. Some CPU architecture even blur the line between hardware registers and stack slots: register windows! Indeed what matters is whether each instruction addresses operands explicitly (registers) or implicitly (top of the stack)

The funny part is all modern CPUs use register renaming (with a 100s deep register file) so they're effectively SSA machines if you squint.

Re: Interesting things about the Lua interpreter (2020)

#26

Post by chc4 »

Lua 5.0 doesn't have the incremental garbage collector, only the original mark-and-sweep collector.

The pdf they based the blog post off of even says the incremental GC is upcoming in 5.1, and you can read through https://www.lua.org/source/5.0/lgc.c.html yourself and see that, unlike the 5.1 version, it only has a single mark list and doesn't use the tri-color scheme.

Re: Interesting things about the Lua interpreter (2020)

#27

Post by wruza »

deathanatos wrote: post #15

I'd love to see upvalues diagrammed as they are represented in memory. It sounds like the stack is perhaps a Stack , where each Frame contains the locals for that stack frame; then a coroutine just needs to keep a pointer to the stack frame it is closing over. (And then, Frame does too, recursively, incase there is more than one scope being closed over.) This would be extremely similar to … most any other language ……

Since all upvalues can be determined at “compile” time (cause they all are locals up the lexical scope), I think they don’t pin stack frames^ for their lifetime, instead they create separate upvalue blocks in advance when corresponding activation records get created. One can also create C closures with upvalues from the stack, e.g. push a string and “close” it for few C functions (lua_pushcclosure, lua_upvaluejoin, luaL_setfuncs). But this string gets popped from the stack, and these functions live on their own and have a shared upvalue, which is not in any stack frame. It would be reasonable to assume from the C API that lua_CFunctions and Lua functions have the same upvalue mechanism, as they are completely interchangeable.

^ I mean internal stack structures, not a public stack, which is fully dynamic and may be cleared without affecting anything, e.g. in lua_CFunction. There’s nothing you can refer to on the stack, the same way you don’t want to refer to a temporary technical `push` in Assembly. Consider this:

  function f()
    local arr = []
    for i = 0, 100000000 do
      table.insert(arr, function ()
        return i
      end)
    end
  end
If `i` was created on the stack on every iteration, it would quickly overflow it, because you can’t just erase previous i, since it was closed.

(For those unaware: `for i …` “creates” a new variable/upvalue at each iteration. Every arr[?]() returns a different value in this example.)

Re: Interesting things about the Lua interpreter (2020)

#28

Post by munificent »

deathanatos wrote: post #15

I'd love to see upvalues diagrammed as they are represented in memory. It sounds like the stack is perhaps a Stack , where each Frame contains the locals for that stack frame; then a coroutine just needs to keep a pointer to the stack frame it is closing over. (And then, Frame does too, recursively, incase there is more than one scope being closed over.) This would be extremely similar to … most any other language ……

> It sounds like the stack is perhaps a Stack,

No, the stack isn't a list of frames where each frame is a separate heap allocated list of slots for values. You're right that that's a very common (but not super fast) implementation technique.

The Lua stack is a single contiguous array of value slots that is used by all call frames. In fact, call frames use overlapping stack windows so that function call arguments don't need to be moved during a call.

That makes implementing closures particularly tricky since when a function returns, its region of the stack is immediately reused by later code.

Upvalues address that by moving just the local variables that are closed over off that contiguous stack array and into the heap. And they do that only when the function where the variable is declared returns so that during that functionn's lifetime, it can be accessed directly from the stack. Also, the compiler is able to manage all this just with a single pass compilation.

It's really clever.

Re: Interesting things about the Lua interpreter (2020)

#29

Post by astrange »

xenadu02 wrote: post #25
ithkuil wrote: post #13

Earlier quoted context omitted.

This. Some CPU architecture even blur the line between hardware registers and stack slots: register windows! Indeed what matters is whether each instruction addresses operands explicitly (registers) or implicitly (top of the stack)

The funny part is all modern CPUs use register renaming (with a 100s deep register file) so they're effectively SSA machines if you squint.

Having so many registers but only a few register names helps because you don’t have to save/restore them all in context switches.

Re: Interesting things about the Lua interpreter (2020)

#30

Post by deathanatos »

wruza wrote: post #27
deathanatos wrote: post #15

I'd love to see upvalues diagrammed as they are represented in memory. It sounds like the stack is perhaps a Stack , where each Frame contains the locals for that stack frame; then a coroutine just needs to keep a pointer to the stack frame it is closing over. (And then, Frame does too, recursively, incase there is more than one scope being closed over.) This would be extremely similar to … most any other language ……

Since all upvalues can be determined at “compile” time (cause they all are locals up the lexical scope), I think they don’t pin stack frames^ for their lifetime, instead they create separate upvalue blocks in advance when corresponding activation records get created. One can also create C closures with upvalues from the stack, e.g. push a string and “close” it for few C functions (lua_pushcclosure, lua_upvaluejoin, l…

For first, I was hoping to omit obvious optimization like splitting unclosed locals into a stack for the simplicity of the conversation, to arrive at a conceptual model for upvalues. That's going to be clearly impossible. That's most of your first paragraph. The Lua C API … almost serves to muddle the issue. (The stack there is … an artifact of the API, really? from a point of view trying to model the language.)

(A corrected version of your counter-example:

  function f()
    local arr = {}
    for i = 0, 5 do
      table.insert(arr, function ()
        return i
      end)
    end
    return arr
  end
I've also changed the constant to make it reasonable to run, without creating an enormous table.)

That's good counter-example, and does run counter to my intuition. (The behavior I describe, e.g., in Python, is different; the closures in Python all close over the same variable.) One might be able to boil this down to instead using "scope" where I was saying "stack frame". (The difference being that for loops in Lua create scopes, vs. Python, where they do not. IMO Lua's is the better of the two options.)

To combine the optimization in the sibling thread (that I was attempting to avoid for simplicity's sake), then the implementation could be a stack of scope-ishes, consisting of the unclosed variables only, and then optionally some structure under GC, containing the closed-over variables (none, if nothing is closed over). The scopes on the stack would need to know of the (potential) GC object holding the closed over variables; a (normal) use of `i` here would have to look up the same `i` as what the closure hold, and as you note, cannot live on the stack.

… this is where, again, I think a diagram would help. I don't know that terms like "upvalue" and "activation record" help build understanding of Lua to new practitioners, and IMO, do more to obscure it. (This isn't your fault, ofc.; these are the terms Lua itself uses for these concepts.)

Post reply on HN