Live data from Hacker News

Zero-copy deserialization in Julia

scattered-thoughts.net

31–34 of 34 posts

Re: Zero-copy deserialization in Julia

#31
post #8
post #4

Cool! Could this be used to make on-disk DataFrames that store arbitrary Julia types? Serializing/deserializing such dataframes with JLD has given me lots of trouble (both too slow and also sometimes the files get corrupted in ways that can't be debugged.)

JuliaDB already does this eg `loadtable("foo.csv", output="foo/")` will store the data in "foo/" in binary form and mmap it.

Thanks for the tip Jamii, and for the nice original post. This is pretty close to what I've been doing, but since I'm storing Julia values rather than numeric data I still to need to convert my data to strings to store in "foo.csv" and then parse back from strings when reading in individual values using `loadtable("foo.csv", output="foo/", colparsers=[parserfortype1, parserfortype2, ...])`. As a parser for nested Julia types is fairly complicated (structs containing dictionaries as fields which contain nested arrays as values which contain ...), it'd be nice to go directly to the binary form used by JuliaDB without converting to/from the string intermediate representation needed by the csv format. Perhaps Blob isn't quite the abstraction needed, but if such a thing exists I'd love to know about it!

Re: Zero-copy deserialization in Julia

#32
post #24
post #23

Earlier quoted context omitted.

Do you mind elaborating?

I am not OP but I'm imagining data-oriented design. So like, things are structured and then serialized - and at some other end deserialized in place ready to read cache-coherently?

Yup pretty much. Zero-copy deserialization implies one large read into a block of memory(or mmap) which forces a specific layout. From there it's straightforward to follow standard data oriented design principals.

Re: Zero-copy deserialization in Julia

#33
post #30
post #22

Earlier quoted context omitted.

Point taken. It would be nice to be able to query my code base to see if unsafe methods have been used (and where) in the executable.

Rust is really good that way. It would be nice to have for sure.

This is a Rust version, btw:

   https://github.com/frankmcsherry/abomonation
It still needs me to opt in to declaring methods as unsafe, rather than automatically surfacing my uses of unsafe. I could definitely "trick users" by not doing that and pretending it always works (it doesn't).
Post reply on HN