Deserializing Binary Data Files in Rust
adventures.michaelfbryan.com
Deserializing Binary Data Files in Rust
1–10 of 23 posts
Re: Deserializing Binary Data Files in Rust
#2I think c struct is missing alignment attribute? This idea has been explored in flatbuffers, capnproto and probably others.
Re: Deserializing Binary Data Files in Rust
#3I think c struct is missing alignment attribute? This idea has been explored in flatbuffers, capnproto and probably others.
C structs are generally assumed to be laid out in the order defined with proper padding automatically inserted (as most platforms at best dislike unaligned accesses). Alignment information should only be necessary if you want to pack the structure or need to apply wider than standard alignment (e.g. need to align a 32 or 64b type to 128 bytes).
The one possible issue is that reading padding is UB (i think). But here there should be no padding: the struct is a pair number of bytes (alignment 1) and a word (alignment 2), which us aligned due to the previous, and the struct is if alignment 2.
Re: Deserializing Binary Data Files in Rust
#4I think c struct is missing alignment attribute? This idea has been explored in flatbuffers, capnproto and probably others.
C structs are generally assumed to be laid out in the order defined with proper padding automatically inserted (as most platforms at best dislike unaligned accesses). Alignment information should only be necessary if you want to pack the structure or need to apply wider than standard alignment (e.g. need to align a 32 or 64b type to 128 bytes). The one possible issue is that reading padding is UB (i think). But here…
yeah C will be order defined in code, rust will reorder to save space unless you attribute it not to. interesting trivia a blind guy wrote the reordering code. if anyone knows his name please post.
Re: Deserializing Binary Data Files in Rust
#5Earlier quoted context omitted.
C structs are generally assumed to be laid out in the order defined with proper padding automatically inserted (as most platforms at best dislike unaligned accesses). Alignment information should only be necessary if you want to pack the structure or need to apply wider than standard alignment (e.g. need to align a 32 or 64b type to 128 bytes). The one possible issue is that reading padding is UB (i think). But here…
yeah C will be order defined in code, rust will reorder to save space unless you attribute it not to. interesting trivia a blind guy wrote the reordering code. if anyone knows his name please post.
The rust structure is defined as repr(C) so there is no reordering.
Re: Deserializing Binary Data Files in Rust
#6I think c struct is missing alignment attribute? This idea has been explored in flatbuffers, capnproto and probably others.
C structs are generally assumed to be laid out in the order defined with proper padding automatically inserted (as most platforms at best dislike unaligned accesses). Alignment information should only be necessary if you want to pack the structure or need to apply wider than standard alignment (e.g. need to align a 32 or 64b type to 128 bytes). The one possible issue is that reading padding is UB (i think). But here…
That's his point, you want to 1-align the struct to disable the padding to ensure the size of the structure is portable.
Re: Deserializing Binary Data Files in Rust
#7For such a problem, I really recommend https://kaitai.io/
From their website:
> Kaitai Struct is a declarative language used to describe various binary data structures, laid out in files or in memory: i.e. binary file formats, network stream packet formats, etc.
They even have a Rust interface: https://github.com/kaitai-io/kaitai_struct_rust_runtime
Re: Deserializing Binary Data Files in Rust
#8Earlier quoted context omitted.
yeah C will be order defined in code, rust will reorder to save space unless you attribute it not to. interesting trivia a blind guy wrote the reordering code. if anyone knows his name please post.
The rust structure is defined as repr(C) so there is no reordering.
Does the C spec actually specify it needs to be in that order? Been awhile since I messed that that sort of thing. I know in practice they typically just put them in the same order and align on some sort of default packing depending on CPU. Usually to minimize on extra instructions. I have eeked out some extra perf on some CPUs and structs just by moving stuff around and making sure it fits into a cacheline. Sometimes the extra instructions are worth it to unpack some vars if you can get the data into a cacheline. But that depends on your data and usage.
Re: Deserializing Binary Data Files in Rust
#9I really wish there was a safe way to initialize structs with arbitrary binary data in rust. Perhaps I'm missing something, but it seems intuitively safe, especially if the struct only contains integer primitives.
Re: Deserializing Binary Data Files in Rust
#10Earlier quoted context omitted.
C structs are generally assumed to be laid out in the order defined with proper padding automatically inserted (as most platforms at best dislike unaligned accesses). Alignment information should only be necessary if you want to pack the structure or need to apply wider than standard alignment (e.g. need to align a 32 or 64b type to 128 bytes). The one possible issue is that reading padding is UB (i think). But here…
yeah C will be order defined in code, rust will reorder to save space unless you attribute it not to. interesting trivia a blind guy wrote the reordering code. if anyone knows his name please post.
Implemented in https://github.com/rust-lang/rust/pull/37429 ; his website seems to be https://ahicks.io/