Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

91–100 of 378 posts

Re: The Byte Order Fiasco

#91

Post by cygx »

themulticaster wrote: post #83

The first example in the article is flawed (or at least misleading). 1) They define a char array (which defaults to signed char, as mentioned in the post), including the value 0x80 which can't be represented in char, resulting in a compiler warning (e.g. in GCC 11.1). The mentioned reason against using unsigned char (that shifting 128 left by 24 places results in UB) is also misleading: I could not reproduce the UB w…

> The mentioned reason against using unsigned char (that shifting 128 left by 24 places results in UB) is also misleading

No, that reasoning is correct. Integer promotions are performed on the operands of a shift expression, meaning the left operand will be promoted to signed int even if it starts out as unsigned char. Trying to shift a byte value with highest bit set by 24 will results in a value not representable as signed int, leading to UB.

Re: The Byte Order Fiasco

#92

Post by st_goliath »

rwmj wrote: post #8

If you can assume GCC or Clang then __builtin_bswap{16,32,64} functions are provided which will be considerably more efficient, less error-prone, and easier to use than anything you can homebrew.

Well, yes. The only thing missing is knowing if you have to swap or not, if you don't want to assume your code will run on little endian systems exclusively.

Or, on Linux and BSD systems at least, you can use the or functions (https://linux.die.net/man/3/endian) and rely on the libc implementation to do the system/compiler detection for you and use an appropriate compiler builtin inside of an inline function instead of bothering to hack something together in your own code.

The article mentions those functions at the bottom, but strangely still recommends hacking up your own macros.

Re: The Byte Order Fiasco

#93

Post by bombcar »

jcadam wrote: post #60
metiscus wrote: post #27

Earlier quoted context omitted.

If memory serves correctly, ada 2012 and beyond has language level support for this. I was working on porting some code from an aviation platform to run on PC and it was all in ada 2005 so we didn't have the benefit of that available.

Same here, Ada2005 for the port. The simulator was originally written in Ada95. Part of what made it even less fun was the data was highly packed and individual fields crossed byte boundaries (these 5 bits are X, the next 4 bits are Y, etc.) :(

Given enough memory it may be worth treating the whole stream internally as a bitstream.

Re: The Byte Order Fiasco

#94

Post by jart »

nly wrote: post #87
jart wrote: post #82

Earlier quoted context omitted.

One of her slides was titled "Stop teaching pointers!" too. My VP back at my old job snapped at me once because I got too excited about the pointer abstractions provided by modern C++. Ever since that day I try to take a more rational approach to writing native code where I consider what it looks like in binary and I've configured my Emacs so it can do what clang.godbolt.org does in a single keystroke.

For the record, she's not really saying people shouldn't learn this low level stuff... just that 'intro to C++' shouldn't be teaching this stuff first The biggest problem with C++ in industry is that people tend to write "C/C++" when it deserves to be recognized as a language in its own right.

One does not simply introduce C++. It's the most insanely hardcore language there is. I wouldn't have stood any chance understanding it had it not been for my gentle introduction with C for several years.

Re: The Byte Order Fiasco

#95

Post by BlueTemplar »

wongarsu wrote: post #56
BlueTemplar wrote: post #36

Earlier quoted context omitted.

We really should have moved to 32 bit bytes when moving to 64 bit words. Would have simplified Unicode considerably.

People were holding off on transitioning because pointers use twice as much space in x64. If bytes had quadrupled in space with x64 we would still be using 32 bit software everywhere

Well, obviously it would have delayed the transition. However you can only go so far with 4Go-limited memory.

And do you have examples of still widely used 8-bit sized data formats ?

Re: The Byte Order Fiasco

#96

Post by bombcar »

tempodox wrote: post #68
mrlonglong wrote: post #34

In an ideal world which endian format would one go for?

I for one would go for big-endian, simply because reading memory dumps and byte blocks in assembly or elsewhere works without mental byte-swapping arithmetics for multi-byte entities. Just out of curiosity, I would be interested in learning why so many CPUs today are little-endian. Is it because it is cheaper / more efficient for processor implementations or is it because “the others do it, so we do it the same way”?

https://stackoverflow.com/questions/5185551/why-is-x86-littl...

It simplifies certain instructions internally. Practically everything is little endian because x86 won.

> And if you think about a serial machine, you have to process all the addresses and data one-bit at a time, and the rational way to do that is: low-bit to high-bit because that’s the way that carry would propagate. So it means that [in] the jump instruction itself, the way the 14-bit address would be put in a serial machine is bit-backwards, as you look at it, because that’s the way you’d want to process it. Well, we were gonna built a byte-parallel machine, not bit-serial and our compromise (in the spirit of the customer and just for him), we put the bytes in backwards. We put the low- byte [first] and then the high-byte. This has since been dubbed “Little Endian” format and it’s sort of contrary to what you’d think would be natural. Well, we did it for Datapoint. As you’ll see, they never did use the [8008] chip and so it was in some sense “a mistake”, but that [Little Endian format] has lived on to the 8080 and 8086 and [is] one of the marks of this family.

Re: The Byte Order Fiasco

#97

Post by mhh__ »

captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

D's machine model does actually assume the hardware, and using the compile time metaprogramming you can pretty much do whatever you want when it comes to bit twiddling - whether that means assembly, flags etc.

Re: The Byte Order Fiasco

#98

Post by themulticaster »

nly wrote: post #23

This is why, in 2021, the mantra that C is a good language for these low level byte twiddling tasks needs to die. Dealing with alignment and endianness properly requires a language that allows you to build abstractions. The following is perfectly well defined in C++, despite looking like almost the same as the original unsafe C: #include #include using namespace boost::endian; unsigned char b[5] = {0x80,0x01,0x02,0x0…

This has nothing to do with C++ because your example only hides the real issue occurring in the blog post example: The unaligned read on the array. Try adding something like

  printf("%08x\n", *((uint32_t*)(b)));
to your example and you'll see that it produces UB as well. The reason there is no UB with big_uint32_t probably is that that struct/class/whatever it is probably redefines its dereferencing operator to perform byte-wise reads.

Godbolt example: https://gcc.godbolt.org/z/seWrb5cz7

Re: The Byte Order Fiasco

#99

Post by raphlinus »

nly wrote: post #79
simias wrote: post #71

Earlier quoted context omitted.

Operator overloading is a mixed blessing though, it can be very convenient but it's also very good at obfuscating what's going on. For instance I'm not familiar with this boost library so I'd have a lot of trouble piecing out what your snippet does, especially since there's no explicit function call besides the printf. Personally if we're going the OOP route I'd much prefer something like Rust's `var.to_be()`, `var.t…

That's fine if whatever type 'var' happens to be is NOT usable as an arithmetic type, otherwise you can easily just forget to call .to_le() or .to_native(), or whatever, and end up with a bug. I don't know Rust, so don't know if this is the case? Boost.Endian actually lets you pick between arithmetic and buffer types. 'big_uint32_buf_t' is a buffer type that requires you to call .value() or do a conversion to an inte…

The idiomatic way to do this in Rust is to use functions like .to_le_bytes(), so you have the u32 (or whatever) on one end and raw bytes (something like [u8; 4]) on the other. It can get slightly tedious if you're doing it by hand, but it's impossible to accidentally forget. If you're doing this kind of thing at scale, like dealing with TrueType fonts (another bastion of big-endian), it's common to reach for derive macros, which automate a great deal of the tedium.

Re: The Byte Order Fiasco

#100

Post by mafuy »

BenoitEssiambre wrote: post #53
tssva wrote: post #13

Earlier quoted context omitted.

Network byte order is big endian so it is far from being pretty much irrelevant these days.

Also, this might be irrelevant at the cpu level, but within a byte, bits are usually displayed most significant bit first, so with little endian you end up with bit order: 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 instead of 15 to 0 This is because little endian is not how humans write numbers. For consistency with little endianness we would have to switch to writing "one hundred and twenty three" as 321

Correct me if I'm wrong, but were the now common numbers not imported in the same order from Arabic, which writes right to left? So numbers were invented in little endian, and we just forgot to translate their order.
Post reply on HN