Zero-copy deserialization in Julia
scattered-thoughts.net
Zero-copy deserialization in Julia
1–10 of 34 posts
Re: Zero-copy deserialization in Julia
#2Awesome article and demonstration! Blob seems to be a pretty nice package as well.
Re: Zero-copy deserialization in Julia
#3That is pretty amazing! I wonder how would it look like if it was written in C. Anyone has some links to "Blobs" implementations in C?
Re: Zero-copy deserialization in Julia
#4Cool! Could this be used to make on-disk DataFrames that store arbitrary Julia types? Serializing/deserializing such dataframes with JLD has given me lots of trouble (both too slow and also sometimes the files get corrupted in ways that can't be debugged.)
Re: Zero-copy deserialization in Julia
#5Curious if making the Julia reader for Apache Arrow support Plasma may be more future friendly:
* https://github.com/ExpandingMan/Arrow.jl
* https://arrow.apache.org/blog/2017/08/08/plasma-in-memory-ob...
Re: Zero-copy deserialization in Julia
#6Curious if making the Julia reader for Apache Arrow support Plasma may be more future friendly: * https://github.com/ExpandingMan/Arrow.jl * https://arrow.apache.org/blog/2017/08/08/plasma-in-memory-ob...
Arrow is basically a dataframe format. It's not really suitable for building data-structures. Eg one of the things Blobs is being used to implement is a Bε-tree (http://supertech.csail.mit.edu/papers/BenderFaJa15.pdf).
Blobs is less a data-format and more a replacement for C's ability to cast arbitrary memory locations to structs.
Re: Zero-copy deserialization in Julia
#7That is pretty amazing! I wonder how would it look like if it was written in C. Anyone has some links to "Blobs" implementations in C?
I've used a protobuf library for Arduino that was zero copy... A lot of callbacks to do something with each new thing that's read.
Re: Zero-copy deserialization in Julia
#8Cool! Could this be used to make on-disk DataFrames that store arbitrary Julia types? Serializing/deserializing such dataframes with JLD has given me lots of trouble (both too slow and also sometimes the files get corrupted in ways that can't be debugged.)
JuliaDB already does this eg `loadtable("foo.csv", output="foo/")` will store the data in "foo/" in binary form and mmap it.
Re: Zero-copy deserialization in Julia
#9That is pretty amazing! I wonder how would it look like if it was written in C. Anyone has some links to "Blobs" implementations in C?
If you allow codegen in advance it looks pretty similar eg https://capnproto.org/cxx.html
Re: Zero-copy deserialization in Julia
#10That kind of reminds me of dumpable [1] library doing a similar thing in C++.