Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Ldump – serialize any Lua data

github.com

Post reply on HN
Print view View original

11–20 of 44 posts

Re: Show HN: Ldump – serialize any Lua data

#11

Post by girvel »

lifthrasiir wrote: post #7
sebstefan wrote: post #6

So it also dumps functions and is able to import them back? Does the function still need to be in memory to be loaded again ("does it just dump the pointer") or can I save it to disk, shut off the interpreter, boot it again and it imports it fine (in which case it somehow dumps them as code...?)? Even in the linked test case on the readme you don't show the output/expectation of the serialization

Functions are apparently serialized as a bytecode dump contained in a self-extracting expression. So everything can indeed be serialized as a Lua expression. Seems that the author also tried to preserve as many upvalues as possible, though I feel that is way more dangerous than I would like.

Yep, that is correct. I think ldump is able to preserve all upvalues, even on edge cases such as "_ENV" and joined upvalues (multiple functions referencing one upvalue). A closure is basically an object with a single method and upvalues as fields -- serialization is straightforward. I think I got it covered, but I would be glad to hear ideas about where the serialization can be unstable.

Re: Show HN: Ldump – serialize any Lua data

#12

Post by myrmidon »

sebstefan wrote: post #6

So it also dumps functions and is able to import them back? Does the function still need to be in memory to be loaded again ("does it just dump the pointer") or can I save it to disk, shut off the interpreter, boot it again and it imports it fine (in which case it somehow dumps them as code...?)? Even in the linked test case on the readme you don't show the output/expectation of the serialization

> can I save it to disk, shut off the interpreter, boot it again and it imports it fine (in which case it somehow dumps them as code...?

Yes, it dumps them as bytecode (probably not compatible between completely different interpreters).

It even preserves debug metadata, so stack traces involving serialized/deserialized functions look right, and still show the original source file.

This is really neat.

Re: Show HN: Ldump – serialize any Lua data

#13

Post by girvel »

lifthrasiir wrote: post #9

Maybe I'm too pedantic but allowing anything to be "deserialized", which equals to "evaluated" here, is not secure. I think it only has to accept a very limited subset of Lua anyway, so you may switch to a non-Lua format which is made easy to parse. That way the library has a total control over what is being evaluated.

This is an interesting thought. Currently, it is unsafe and intended to load only the files you trust. I should definitely include a warning into README.

Overall, it would be nice to make it safer. I don't think switching to non-Lua format would make it safer, because it is intended to serialize functions too, which can have arbitrary code even if everything else would be stored as data. Maybe it is possible to make a function like `ldump.safe_load` restricting `load`'s environment, so it wouldn't have access to debug/os/io modules.

Re: Show HN: Ldump – serialize any Lua data

#15

Post by girvel »

myrmidon wrote: post #12
sebstefan wrote: post #6

So it also dumps functions and is able to import them back? Does the function still need to be in memory to be loaded again ("does it just dump the pointer") or can I save it to disk, shut off the interpreter, boot it again and it imports it fine (in which case it somehow dumps them as code...?)? Even in the linked test case on the readme you don't show the output/expectation of the serialization

> can I save it to disk, shut off the interpreter, boot it again and it imports it fine (in which case it somehow dumps them as code...? Yes, it dumps them as bytecode (probably not compatible between completely different interpreters). It even preserves debug metadata, so stack traces involving serialized/deserialized functions look right , and still show the original source file. This is really neat.

Thank you, it is really nice to hear. Though, I have to give credit to Lua's standard library -- the basic function serialization (without upvalues) is implemented there as `string.dump`.

Re: Show HN: Ldump – serialize any Lua data

#17

Post by lifthrasiir »

girvel wrote: post #13
lifthrasiir wrote: post #9

Maybe I'm too pedantic but allowing anything to be "deserialized", which equals to "evaluated" here, is not secure. I think it only has to accept a very limited subset of Lua anyway, so you may switch to a non-Lua format which is made easy to parse. That way the library has a total control over what is being evaluated.

This is an interesting thought. Currently, it is unsafe and intended to load only the files you trust. I should definitely include a warning into README. Overall, it would be nice to make it safer. I don't think switching to non-Lua format would make it safer, because it is intended to serialize functions too, which can have arbitrary code even if everything else would be stored as data. Maybe it is possible to make…

Yeah, you would need an allowlist for functions. Using bytecode would make it much harder, I haven't given deep thought yet.

Re: Show HN: Ldump – serialize any Lua data

#19

Post by nicoloren »

JourneyJourney wrote: post #16

I'm afraid I spent too much time with LUA lately and fell in love with its simplicity. Kinda hard to go back to OOP after that.

Same for me, I used Lua for a desktop software for a client and I enjoyed it a lot!

I'm thinking of starting to dev a game with LOVE2D just to have an excuse to use Lua.

Post reply on HN