Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

81–90 of 378 posts

Re: The Byte Order Fiasco

#81

Post by gpanders »

BenoitEssiambre wrote: post #53
tssva wrote: post #13

Earlier quoted context omitted.

Network byte order is big endian so it is far from being pretty much irrelevant these days.

Also, this might be irrelevant at the cpu level, but within a byte, bits are usually displayed most significant bit first, so with little endian you end up with bit order: 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 instead of 15 to 0 This is because little endian is not how humans write numbers. For consistency with little endianness we would have to switch to writing "one hundred and twenty three" as 321

It might not be how humans write numbers but it is consistent with how we think about numbers in a base system.

123 = 3x10^0 + 2x10^1 + 1x10^2

So if you were to go and label each digit in 123 with the power of 10 it represents, you end up with little endian ordering (eg the 3 has index 0 and the 1 has index 2). This is why little endian has always made more sense to me, personally.

Re: The Byte Order Fiasco

#82

Post by jart »

nly wrote: post #58
jart wrote: post #46

Earlier quoted context omitted.

C is perfect for these problems. I like teaching the endian serialization problem because it broaches so many of the topics that are key to understanding C/C++ in general. Even if we choose to spend the majority of our time plumbing together functions written by better men, it's nice to understand how the language is defined so we could write those functions, even if we don't need to.

For sure, it's a good way to teach that C is insufficient to deal with even the simplest of tasks. Unfortunately teaching has a bad habit of becoming practice, no matter how good the intention. With regard to teaching C++ specifically I tend to agree with this talk: CppCon 2015 - Kate Gregory “Stop Teaching C": https://www.youtube.com/watch?v=YnWhqhNdYyk

One of her slides was titled "Stop teaching pointers!" too. My VP back at my old job snapped at me once because I got too excited about the pointer abstractions provided by modern C++. Ever since that day I try to take a more rational approach to writing native code where I consider what it looks like in binary and I've configured my Emacs so it can do what clang.godbolt.org does in a single keystroke.

Re: The Byte Order Fiasco

#83

Post by themulticaster »

The first example in the article is flawed (or at least misleading).

1) They define a char array (which defaults to signed char, as mentioned in the post), including the value 0x80 which can't be represented in char, resulting in a compiler warning (e.g. in GCC 11.1).

The mentioned reason against using unsigned char (that shifting 128 left by 24 places results in UB) is also misleading: I could not reproduce the UB when changing the array to unsigned char. Perhaps the author meant leaving the array defined as signed char, but casting the signed chars to unsigned before shifting. That indeed results in UB, but I don't see why you would define the array as signed in the first place.

2) The cause for the undefined behavior isn't the bswap_32, rather it's because they try reading an uint32_t value from a char array, where b[0] is not aligned on a word boundary.

There is no need at all do redefine bswap. The simple solution would be to use an unsigned char array instead of a char array and just reading the values byte-wise.

Of course C has its footguns and warts and so on, but there is no need to dramatize it this much in my opinion.

I've prepared a Godbolt example to better explain the arguments mentioned above: https://godbolt.org/z/Y1EWK6e17

Edit: To add to point 2) above: Another way to avoid the UB (in this specific case) would be to add __attribute__ ((aligned (4))) to the definition of b. In that case, even reading the array as a single uint32_t works as expected since the access is aligned to a word boundary.

Obviously, you can't expect any random (unsigned char) pointer to be aligned on a word boundary. Therefore, it is still necessary to read the uint32_t byte by byte.

Re: The Byte Order Fiasco

#84

Post by nly »

cbmuser wrote: post #78
nly wrote: post #55

Earlier quoted context omitted.

No, the fact that this can be done in a library and looks like a native language feature demonstrates the power of C++ as a language. This example is demonstrating: - First class treatment of user (or library) defined types - Operator overloading - The fact that it produces fast machine code. Try changing big_uint32_t to regular uint32_t to see how this changes. When you use the later ubsan will introduce a trap for…

You are still casting one pointer type into another which can result in unaligned access. If you need to change byte orders, you should use library to achieve that.

Boost.Endian is the library here and this code is safe because the big_uint32_t type has an alignment requirement of 1 byte.

This is why ubsan is silent and not even injecting a check in to the compiled code.

You can check the alignment constraints with static_assert (something else you can't do in standard C): https://gcc.godbolt.org/z/KTcf9ax6r

Re: The Byte Order Fiasco

#85

Post by jart »

pabs3 wrote: post #17
londons_explore wrote: post #15

Remember how we used to have machines with a 7 bit byte? And everything was written to handle either 6, 7, or 8 bit bytes? And now we've settled on all machines being 8 bit bytes, and programmers no longer have to worry about such details? Is it time to do the same for big endian machines? Is it time to accept that all machines that matter are little endian, and the extra effort keeping everything portable to big end…

IBM is going to be pretty annoyed when your code doesn't work on their mainframes.

In my experience IBM does the right thing and sends patches rather than asking us to fix their problems for them, and I respect them for that reason, even if it's a tiny burden to review those changes.

However endianness isn't just about supporting IBM. Modern compilers will literally break your code if you alias memory using a type wider than char. It's illegal per the standard. In the past compilers would simply not care and say, oh the architecture permits unaligned reads so we'll just let you do that. Not anymore. Modern GCC and Clang force your code to conform to the abstract standard definition rather than the local architecture definition.

It's also worth noting that people think x86 architecture permits unaligned reads but that's not entirely true. For example, you can't do unaligned read-ahead on C strings, because in extremely rare cases you might cross a page boundary that isn't defined and trigger a segfault.

Re: The Byte Order Fiasco

#86

Post by MrBuddyCasino »

captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

> There is a huge mismatch between the assumptions of the C spec and actual machine code.

People like to say „C is close to the metal“. Really not true at all anymore.

Re: The Byte Order Fiasco

#87

Post by nly »

jart wrote: post #82
nly wrote: post #58

Earlier quoted context omitted.

For sure, it's a good way to teach that C is insufficient to deal with even the simplest of tasks. Unfortunately teaching has a bad habit of becoming practice, no matter how good the intention. With regard to teaching C++ specifically I tend to agree with this talk: CppCon 2015 - Kate Gregory “Stop Teaching C": https://www.youtube.com/watch?v=YnWhqhNdYyk

One of her slides was titled "Stop teaching pointers!" too. My VP back at my old job snapped at me once because I got too excited about the pointer abstractions provided by modern C++. Ever since that day I try to take a more rational approach to writing native code where I consider what it looks like in binary and I've configured my Emacs so it can do what clang.godbolt.org does in a single keystroke.

For the record, she's not really saying people shouldn't learn this low level stuff... just that 'intro to C++' shouldn't be teaching this stuff first

The biggest problem with C++ in industry is that people tend to write "C/C++" when it deserves to be recognized as a language in its own right.

Re: The Byte Order Fiasco

#88

Post by bombcar »

kstenerud wrote: post #43
bregma wrote: post #7

Earlier quoted context omitted.

And which is the correct byte ordering, pray tell?

Big endian is easier for humans to read when looking at a memory dump, but little endian has many useful features in binary encoding schemes due to the low byte being first. I used to like big endian more, but after deep investigation I now prefer little endian for any encoding schemes.

Couldn’t encoding systems be redone with emphasis on the high-order bits? Or is the assumption that the values are clustered in the low bits?

Re: The Byte Order Fiasco

#89

Post by mytailorisrich »

amelius wrote: post #3

Byte order is one of the great unnecessary historical fuck ups in computing. A similar one is that signedness of char is machine dependent. It's typically signed on Intel and unsigned on ARM. Sigh!

I don't think it's a fuck up, rather I think it was unavoidable: Both ways are equally valid and when the time came to make the decision, some people decided one way, some people decided the other way.

Re: The Byte Order Fiasco

#90

Post by BlueTemplar »

ChrisSD wrote: post #48
BlueTemplar wrote: post #36

Earlier quoted context omitted.

We really should have moved to 32 bit bytes when moving to 64 bit words. Would have simplified Unicode considerably.

Not really. Unicode is a variable width abstract encoding; a single character can be made up of multiple code points. For Unicode, 32-bit bytes would be an incredibly wasteful in memory encoding.

One byte = one "character" makes for much easier programming.

Text generally uses a small fraction of memory and storage these days.

Post reply on HN