Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fallacy

commandcenter.blogspot.com

Post reply on HN
Print view View original

21–30 of 61 posts

Re: The Byte Order Fallacy

#21

Post by iscoelho »

If you are using C/C++ for any new app, there is a possibility you are writing code that has a performance requirement.

- mmap/io_uring/drivers and additional "zero-copy" code implementations require consideration about byte order.

- filesystems, databases, network applications can be high throughput and will certainly benefit from being zero-copy (with benefits anywhere from +1% to +2000% in performance.)

This is absolutely not "premature optimization." If you're a C/C++ engineer, you should know off the top of your head how many cycles syscalls & memcpys cost. (Spoiler: They're slow.) You should evaluate your performance requirements and decide if you need to eliminate that overhead. For certain applications, if you do not meet the performance requirements, you cannot ship.

Re: The Byte Order Fallacy

#23

Post by AlotOfReading »

iscoelho wrote: post #21

If you are using C/C++ for any new app, there is a possibility you are writing code that has a performance requirement. - mmap/io_uring/drivers and additional "zero-copy" code implementations require consideration about byte order. - filesystems, databases, network applications can be high throughput and will certainly benefit from being zero-copy (with benefits anywhere from +1% to +2000% in performance.) This is ab…

A memcpy should not be slow. It should be nearly as fast as generic memory copying can be. Most of the time you shouldn't even hit the actual function, but instead a bit of code generated by the compiler that does exactly the copy you need.

Re: The Byte Order Fallacy

#24

Post by nightowl_games »

AlotOfReading wrote: post #23
iscoelho wrote: post #21

If you are using C/C++ for any new app, there is a possibility you are writing code that has a performance requirement. - mmap/io_uring/drivers and additional "zero-copy" code implementations require consideration about byte order. - filesystems, databases, network applications can be high throughput and will certainly benefit from being zero-copy (with benefits anywhere from +1% to +2000% in performance.) This is ab…

A memcpy should not be slow. It should be nearly as fast as generic memory copying can be. Most of the time you shouldn't even hit the actual function, but instead a bit of code generated by the compiler that does exactly the copy you need.

Yeah ive always been blown away by how fast memcpy is. I'm guessing the OP is from a different world of engineering than I am.

Re: The Byte Order Fallacy

#25

Post by fracus »

I don't like these ambiguous titles. From the title I thought I was going to read that byte order doesn't matter when in fact the title should be "a computer's byte order is irrelevant to high-level languages". At least, state the fallacy in unambiguous terms one sentence right away. In any case, was an interesting read.

Re: The Byte Order Fallacy

#26

Post by wmf »

He's right that you shouldn't use ifdefs, but I think a macro like le32toh() is far clearer and more concise than a bunch of shifts and ors.

Also, a lot of comments in this thread have nothing to do with the article and appear to be responses to some invisible strawman.

Re: The Byte Order Fallacy

#27

Post by ddingus »

fracus wrote: post #25

I don't like these ambiguous titles. From the title I thought I was going to read that byte order doesn't matter when in fact the title should be "a computer's byte order is irrelevant to high-level languages". At least, state the fallacy in unambiguous terms one sentence right away. In any case, was an interesting read.

I came here to write the same. I learned a thing or two about how higher level languages work.

Two areas I find it does matter: Assembly language where bytes are parsed or sorted, or transformed in some way by code that writes words

, and

binary file representations when written on a little endian machine and read by a big endian machine.

Re: The Byte Order Fallacy

#28

Post by benlivengood »

The other case where it matters is SIMD instructions where you're serializing or deserializing multiple fields at once, but the SIMD operations are usually architecture specific to begin with and so if you shuffle bytes into and out of the native packed formats it will be specific to the endianness of the native packed format, and then you can forget about byte order outside of those shuffle transformations.

Re: The Byte Order Fallacy

#29

Post by wmf »

edflsafoiewq wrote: post #19
Laremere wrote: post #10

This is a reasonable way to do things, and I've used it before. However I just used Zig's method here, and like it a lot: https://ziglang.org/documentation/master/std/#std.io.Reader.... Given a reader (file, network, buffers can all be turned into readers), you can call readInt. It takes the type you want, and the endianess of the encoding. It's easy to write, self documents, and it's highly efficient.

If we're talking about a single int, the way you do it doesn't matter, just wrap it up in a readInt function. But if we're talking about a struct or an array, if you're byte-order aware you can do things like memcpy the whole thing around that you couldn't do by assembling it out of individual readInt calls.

It's probably faster to memcpy the thing then "swap" each element (the swaps may be no-ops under the hood). This should be portable and fast.
Post reply on HN