Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fallacy

commandcenter.blogspot.com

Post reply on HN
Print view View original

11–20 of 61 posts

Re: The Byte Order Fallacy

#11

Post by GMoromisato »

rwmj wrote: post #4

Unless you're dealing with binary data in which case byte order matters very much and if you forget to convert it you're causing a world of pain for someone. He even has an example where he just pushes the problem off to someone else "if the people at Adobe wrote proper code to encode and decode their files" , yeah hope they weren't ignoring byte order issues.

The article's point is that the machine's byte order doesn't matter. The byte order of a data stream of course matters, but they show a way to load a binary data stream without worrying about the machine's byte order.

That key insight is that people shouldn't try to optimize the case where the data stream's byte order happens to match the machine's byte order. That's both premature optimization and a recipe for bugs. Just don't worry about that case.

Load binary data one byte at a time and use shifts and ORs to compose the larger unit based on the data's byte order. That's 100% portable without any #ifdefs for the machine's byte order.

Re: The Byte Order Fallacy

#12

Post by GMoromisato »

LegionMammal978 wrote: post #9

IME, there's one big thing that often keeps my programs from being unaffected by byte order: wanting to quickly splat data structures into and out of files, pipes, and sockets, without having to encode or decode each element one-by-one. The only real way to make this endian-independent is to have byte-swapping accessors for everything when it's ultimately produced or consumed, but adding all the code for that is very…

I think the article's author would say that loading data "without having to encode or decode each element" is premature optimization and more likely to have bugs. I tend to agree.

Re: The Byte Order Fallacy

#15

Post by dwattttt »

GMoromisato wrote: post #12
LegionMammal978 wrote: post #9

IME, there's one big thing that often keeps my programs from being unaffected by byte order: wanting to quickly splat data structures into and out of files, pipes, and sockets, without having to encode or decode each element one-by-one. The only real way to make this endian-independent is to have byte-swapping accessors for everything when it's ultimately produced or consumed, but adding all the code for that is very…

I think the article's author would say that loading data "without having to encode or decode each element" is premature optimization and more likely to have bugs. I tend to agree.

The optimisation the parent is referring to is development time/effort; if the alternative to dumping a structure to a file is to hand roll your serialiser/deserialiser, that's a slower & probably more error prone approach (depending on the context).

Re: The Byte Order Fallacy

#16

Post by ajross »

wiredfool wrote: post #13

Unless you’re writing code to decode image file formats.

No, same deal. The article argues that you should write portable code based on the ordered bytes in an external format, as that's guaranteed to be a machine-independent thing (i.e. it's stored on disk in exactly one way). Same is true for image files as 2-byte wchar file as zip files, yada yada.

It's true as far as it goes, but (1) it leans very heavily on the compiler understanding what you're doing and "un-portabilifying" your code when the native byte order matches the file format and (2) it presumes you're working with pickled "file" formats you "stream" in via bytes and not e.g. on memory mapped regions (e.g. network packets!) that want naturally to be inspected/modified in place.

It's fine advice though for the 90% of use cases. The author is correct that people tend to tie themselves into knots needlessly over this stuff.

Re: The Byte Order Fallacy

#17

Post by sobellian »

GMoromisato wrote: post #12
LegionMammal978 wrote: post #9

IME, there's one big thing that often keeps my programs from being unaffected by byte order: wanting to quickly splat data structures into and out of files, pipes, and sockets, without having to encode or decode each element one-by-one. The only real way to make this endian-independent is to have byte-swapping accessors for everything when it's ultimately produced or consumed, but adding all the code for that is very…

I think the article's author would say that loading data "without having to encode or decode each element" is premature optimization and more likely to have bugs. I tend to agree.

Maybe if you hand-roll the struct layout, but if you use something like flatbuffers I doubt you would see many more bugs - and flatbuffers will take care of endian swaps as necessary without you needing to think about it.

Re: The Byte Order Fallacy

#18

Post by skybrian »

dwattttt wrote: post #15
GMoromisato wrote: post #12

Earlier quoted context omitted.

I think the article's author would say that loading data "without having to encode or decode each element" is premature optimization and more likely to have bugs. I tend to agree.

The optimisation the parent is referring to is development time/effort; if the alternative to dumping a structure to a file is to hand roll your serialiser/deserialiser, that's a slower & probably more error prone approach (depending on the context).

The article makes an argument that the hand-rolled solution is less buggy, if you approach it the right way.

For complicated data structures, it's probably best to use a library that serializes to a common standard. (For example, protocol buffers or JSON.)

But I think the article assumes you don't get to choose the protocol, so it probably has to be hand-written by someone.

Re: The Byte Order Fallacy

#19

Post by edflsafoiewq »

Laremere wrote: post #10

This is a reasonable way to do things, and I've used it before. However I just used Zig's method here, and like it a lot: https://ziglang.org/documentation/master/std/#std.io.Reader.... Given a reader (file, network, buffers can all be turned into readers), you can call readInt. It takes the type you want, and the endianess of the encoding. It's easy to write, self documents, and it's highly efficient.

If we're talking about a single int, the way you do it doesn't matter, just wrap it up in a readInt function.

But if we're talking about a struct or an array, if you're byte-order aware you can do things like memcpy the whole thing around that you couldn't do by assembling it out of individual readInt calls.

Re: The Byte Order Fallacy

#20

Post by butterisgood »

AstralStorm wrote: post #7

Really except for the networking (including say Bluetooth) nobody is big endian anymore. So how about just don't leak that thing from the network layer. And do not define any data format to be big endian anymore. Deine it as little endian (do not leave it undefined) and everyone will be happy.

I think both SMB and 9p (Plan 9 resource sharing/file system protocol) are little endian.

So it's not even all networking... and "network byte order" will mess you up.

Post reply on HN