Live data from Hacker News

The Byte Order Fiasco

justine.lol

161–170 of 378 posts

Re: The Byte Order Fiasco

#161

Earlier quoted context omitted.

Also, this might be irrelevant at the cpu level, but within a byte, bits are usually displayed most significant bit first, so with little endian you end up with bit order: 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 instead of 15 to 0 This is because little endian is not how humans write numbers. For consistency with little endianness we would have to switch to writing "one hundred and twenty three" as 321

It might not be how humans write numbers but it is consistent with how we think about numbers in a base system. 123 = 3x10^0 + 2x10^1 + 1x10^2 So if you were to go and label each digit in 123 with the power of 10 it represents, you end up with little endian ordering (eg the 3 has index 0 and the 1 has index 2). This is why little endian has always made more sense to me, personally.

I don't know, when we write in general, we tend to write the most significant stuff first so you lose less information if you stop early. Even numbers we truncate twelve millions instead of something like twelve millions, zero thousand zero hundreds and 0.

Re: The Byte Order Fiasco

#162
post #71
post #55

Earlier quoted context omitted.

No, the fact that this can be done in a library and looks like a native language feature demonstrates the power of C++ as a language. This example is demonstrating: - First class treatment of user (or library) defined types - Operator overloading - The fact that it produces fast machine code. Try changing big_uint32_t to regular uint32_t to see how this changes. When you use the later ubsan will introduce a trap for…

Operator overloading is a mixed blessing though, it can be very convenient but it's also very good at obfuscating what's going on. For instance I'm not familiar with this boost library so I'd have a lot of trouble piecing out what your snippet does, especially since there's no explicit function call besides the printf. Personally if we're going the OOP route I'd much prefer something like Rust's `var.to_be()`, `var.t…

Ironically, it was proposed not so long ago to deprecate to_be/to_le in favour of to_be_bytes/to_le_bytes, since the former conflate abstract values with bit representations.

Re: The Byte Order Fiasco

#163
post #6

Ubsan should default on. If people don't like it, then they should be made turn it off with a switch, so at least it's more likely to be run than not run. Could save a huge amount of time debugging when compilers or architecture changes. Without it, I'd say many a programmer would be caught by these subtleties in the standard. Coming from a HW background (Verilog) I'd more naturally default to masking and shifting wh…

> Ubsan should default on > Could save a huge amount of time debugging when compilers or architecture changes. I'm assuming we come from very different backgrounds, but it's not clear to me how switching compilers or architectures is so common that hardening code against it by default is appropriate. I would think that switching compilers or architectures is generally done very deliberately, so instrumenting code wit…

Changing compilers is a pretty regular thing IMHO; I use the compiler that comes with the OS and let's assume a yearly OS release cycle. Most of those will contain at least some changes to the compiler.

I don't really want to have to take that yearly update to go through and review (and presumablu fix) all the UB that has managed to sneak in over the year. It would be better to have avoided putting it in.

Re: The Byte Order Fiasco

#165
post #96

Earlier quoted context omitted.

I for one would go for big-endian, simply because reading memory dumps and byte blocks in assembly or elsewhere works without mental byte-swapping arithmetics for multi-byte entities. Just out of curiosity, I would be interested in learning why so many CPUs today are little-endian. Is it because it is cheaper / more efficient for processor implementations or is it because “the others do it, so we do it the same way”?

https://stackoverflow.com/questions/5185551/why-is-x86-littl... It simplifies certain instructions internally. Practically everything is little endian because x86 won. > And if you think about a serial machine, you have to process all the addresses and data one-bit at a time, and the rational way to do that is: low-bit to high-bit because that’s the way that carry would propagate. So it means that [in] the jump instr…

And does middle endian even exist?

Re: The Byte Order Fiasco

#166
post #24
post #3

Byte order is one of the great unnecessary historical fuck ups in computing. A similar one is that signedness of char is machine dependent. It's typically signed on Intel and unsigned on ARM. Sigh!

the greatest of all is lisp not being the most mainstream language, and we can only blame the lisp companies for this fiasco. in an ideal world we all would be using a lisp with parametric polymorphism. from highest level abstractions to machine level, all in one language.

[deleted]

Re: The Byte Order Fiasco

#167

FWIW there is a on various BSDs that contains "beXXtoh", "leXXtoh", "htobeXX", "htoleXX" where XX is a number of bits (16, 32, 64). That header is also available on Linux, but glibc (and compatible libraries) named it instead. See: man 3 endian ( https://linux.die.net/man/3/endian ) Of course it gets a bit hairier if the code is also supposed to run on other systems. MacOS has OSSwapHostToLittleIntXX, OSSwapLittleToH…

At least historically windows have had big-endian versions as both SPARC and Itanium use big endian.

Re: The Byte Order Fiasco

#168
post #147
post #3

Byte order is one of the great unnecessary historical fuck ups in computing. A similar one is that signedness of char is machine dependent. It's typically signed on Intel and unsigned on ARM. Sigh!

Why is it an issue any more than say, order of fields in a struct is an issue? In one case you read bytes off the disk by doing ((b[0] << 8) | b[1]) (or equivalent), with the order reversed the other way around. Any application-level (say, not a compiler, debugger, etc) program should not even need to know the native byte order, it should only need to know the encoding that the file it’s trying to read used.

> order of fields in a struct

This is defined in C to be the order the fields are declared in.

Re: The Byte Order Fiasco

#169
post #84
post #78

Earlier quoted context omitted.

You are still casting one pointer type into another which can result in unaligned access. If you need to change byte orders, you should use library to achieve that.

Boost.Endian is the library here and this code is safe because the big_uint32_t type has an alignment requirement of 1 byte. This is why ubsan is silent and not even injecting a check in to the compiled code. You can check the alignment constraints with static_assert (something else you can't do in standard C): https://gcc.godbolt.org/z/KTcf9ax6r

C11 has static_assert: https://gcc.godbolt.org/z/E3bGc95o3

Is also has _Generic() so you can roll up a family of endianness conversion functions and safely change types without blowing up somewhere else with a hardcoded conversion routine.

Re: The Byte Order Fiasco

#170
post #156

Earlier quoted context omitted.

In Rust, [u8; N] and &[u8] are both primitive types, and not abstractions. It's possible to create an abstraction around either (the former even more so now with const generics), but that's not necessary. It's also possible to use "extension traits" to add methods, even to existing and built-in types[1]. I'm not sure about a 3 byte big endian integer. I mean, that's going to compile down to some combination of shifti…

I agree then that in Rust you could make something consistent. I think there's no need for explicit shifts. You need to memcpy anyway to deal with alignment issues, so you may as well just copy in to the last 3 bytes of a zero-initialized, big endian, 32bit uint. https://gcc.godbolt.org/z/jEnsW8WfE

That's just constant folding. Here's what it looks like when you actually need to go to memory:

https://gcc.godbolt.org/z/9qGqh6M1E

And I think we're on the same page, it should be possible to get similar results in Rust.

Post reply on HN