Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

251–260 of 378 posts

Re: The Byte Order Fiasco

#251

Post by jart »

pm215 wrote: post #238
captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

So in your 'machine model is the physical machine' flavour, should "I cast an unaligned pointer to a byte array to int32_t and deref" on SPARC (a) do a bunch of byte-load-and-shift-and-OR or (b) emit a simple word load which segfaults? If the former, it's not what the physical machine does, and if the latter, then you still need to write the code as "some portable other thing". Which is to say that the spec's UB here…

Endianness doesn't matter though, for the reasons Rob Pike explained. For example, the bits inside each byte have an endianness probably inside the CPU but they're not addressable so no one thinks about that. The brilliance of Rob Pike's recommendation is that it allows our code to be byte order agnostic for the same reasons our code is already bit order agnostic.

I agree about bsf/bsr/popcnt. I wish ASCII had more punctuation marks because those operations are as fundamental as xor/and/or/shl/shr/sar.

Re: The Byte Order Fiasco

#252

Post by froh »

infradig wrote: post #249
Animats wrote: post #207

Rust gets this right. These primitives are available for all the numeric types. u32::from_le_byte(bytes) // u32 from 4 bytes, little endian u32::from_be_byte(bytes) // u32 from 4 bytes, big endian u32::to_le_bytes(num) // u32 to 4 bytes, little endian u32::to_be_bytes(num) // u32 to 4 bytes, big endian This was very useful to me recently as I had to write the marshaling and un-marshaling for a game networking format…

There are equivalent functions in C too. The point of the article is about not using them. So how would you implement the above functions in Rust would be more pertinent.

isnt the point to be careful when implementing them? so the compiler detects the intention to byteswap?

when we ported little endian x86 Linux to the big endian mainframe we sprinkled hton/ntoh all over the place, happily so. they are the way to go and they should be implemented properly, not be replaced by a homegrown version.

all that said, I'm surprised 64bit htonll and ntohll are not standard yet. anybody knows why?

Re: The Byte Order Fiasco

#253

Post by froh »

captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

you could instead simply use hton/ntoh and trust the library properly does The Right Thing tm

Re: The Byte Order Fiasco

#254

Post by froh »

ok123456 wrote: post #185
nly wrote: post #23

This is why, in 2021, the mantra that C is a good language for these low level byte twiddling tasks needs to die. Dealing with alignment and endianness properly requires a language that allows you to build abstractions. The following is perfectly well defined in C++, despite looking like almost the same as the original unsafe C: #include #include using namespace boost::endian; unsigned char b[5] = {0x80,0x01,0x02,0x0…

Or just use the functions in to convert from host to network byteorder?

this! use hton/ntoh and be happy.

nitpick: the 64bit versions are not fully available yet, htonll, ntohll

Re: The Byte Order Fiasco

#255

Post by hermitdev »

edflsafoiewq wrote: post #250
BeeOnRope wrote: post #244

Earlier quoted context omitted.

How do you detect if a byte swap is needed? I.e. wether the (fixed) wire endianness matches the current platform endianness?

Ie how do you know the target's endianness? C++20 added std::endian. Otherwise you can use a macro like this one from SDL https://github.com/libsdl-org/SDL/blob/9dc97afa7190aca5bdf92...

There have been CPU architectures where the endianness at compile time isn't necessarily sufficient. I forget which, maybe it was DEC Alpha, where the CPU could flip back and forth? I can't recall if it was a "choose at boot" or a per process change.

Re: The Byte Order Fiasco

#256

Post by froh »

jart wrote: post #85
pabs3 wrote: post #17

Earlier quoted context omitted.

IBM is going to be pretty annoyed when your code doesn't work on their mainframes.

In my experience IBM does the right thing and sends patches rather than asking us to fix their problems for them, and I respect them for that reason, even if it's a tiny burden to review those changes. However endianness isn't just about supporting IBM. Modern compilers will literally break your code if you alias memory using a type wider than char. It's illegal per the standard. In the past compilers would simply no…

yes IBM provided asm for s390 hton ntoh, and "all we had to do" for mainframe Linux was patch x86 only packages to use hton ntoh when they persisted binary data. for the kernel IBM did it on their own, contributing mainline, for userland suse did it, grabbing some patches from japanese turbolinux, and then red hat grabbed the patches from turbo and suse, and together we got them mainline lol. and PPC then just piggybacked on top of that effort.

Re: The Byte Order Fiasco

#257

Post by froh »

BenoitEssiambre wrote: post #53
tssva wrote: post #13

Earlier quoted context omitted.

Network byte order is big endian so it is far from being pretty much irrelevant these days.

Also, this might be irrelevant at the cpu level, but within a byte, bits are usually displayed most significant bit first, so with little endian you end up with bit order: 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 instead of 15 to 0 This is because little endian is not how humans write numbers. For consistency with little endianness we would have to switch to writing "one hundred and twenty three" as 321

that's why little endian == broken endian

said a friend who also quips: "never trust a computer you can lift"

Re: The Byte Order Fiasco

#259

Post by froh »

bigbillheck wrote: post #150

I just use ntohl/htonl like a civilized person. (Yes, the article mentions those, but they've been standard for decades).

what's the best practice for 64bit values these days? is htonll ntohll widely available yet?

Re: The Byte Order Fiasco

#260

Post by pornel »

captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

Of course nobody wants C to backstab them with UB, but at the same time programmers want compilers to generate optimal code. That's the market pressure that forces optimizers to be so aggressive. If you can accept less optimized code, why aren't you using tcc?

The idea of C that "just" does a straightforward machine translation breaks down almost immediately. For example, you'd want `int` to just overflow instead of being UB. But then it turns out indexing `arr[i]` can't use 64-bit memory addressing modes, because they don't overflow like a 32-bit int does. With UB it doesn't matter, but a "straightforward C" would emit unnecessary separate 32-bit mul/shift instructions.

https://gist.github.com/rygorous/e0f055bfb74e3d5f0af20690759...

Post reply on HN