Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

241–250 of 378 posts

Re: The Byte Order Fiasco

#241

Post by goldenkey »

MrBuddyCasino wrote: post #86
captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

> There is a huge mismatch between the assumptions of the C spec and actual machine code. People like to say „C is close to the metal“. Really not true at all anymore.

Actually, it is true - which is why endian is a problem in the first place. ASM code is different when written for little endian vs big endian. Access patterns are positively offset instead of negatively.

A language that does the same things regardless of endianness would not have pointer arithmetic. That is not ASM and not C.

Re: The Byte Order Fiasco

#242

Post by wnoise »

ajross wrote: post #74
captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

> There is a huge mismatch between the assumptions of the C spec and actual machine code. Right, which is why the kind of UB pedantry in the linked article is hurting and not helping. Cranky old man perspective here: Folks: the fact that compilers will routinely exploit edge cases in undefined behavior in the language specification to miscompile obvious idiomatic code is a terrible bug in the compilers . Period. And…

I read this, and go "yes, yes, yes", and then "NO!".

Shifts and ors really is the sanest and simplest way to express "assembling an integer from bytes". Masking is _a_ way to deal with the current C spec which has silly promotion rules. Unsigned everything is more fundamental than signed.

Re: The Byte Order Fiasco

#243

Post by jart »

ajross wrote: post #74
captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

> There is a huge mismatch between the assumptions of the C spec and actual machine code. Right, which is why the kind of UB pedantry in the linked article is hurting and not helping. Cranky old man perspective here: Folks: the fact that compilers will routinely exploit edge cases in undefined behavior in the language specification to miscompile obvious idiomatic code is a terrible bug in the compilers . Period. And…

I agree but language of the standard very unambiguously lets them do it. Quoth X3.159-1988

     * Undefined behavior --- behavior, upon use of a nonportable or
       erroneous program construct, of erroneous data, or of
       indeterminately-valued objects, for which the Standard imposes no
       requirements.  Permissible undefined behavior ranges from ignoring the
       situation completely with unpredictable results, to behaving during
       translation or program execution in a documented manner characteristic
       of the environment (with or without the issuance of a diagnostic
       message), to terminating a translation or execution (with the issuance
       of a diagnostic message).
In the past compilers "behaved during translation or program execution in a documented manner characteristic of the environment" and now they've decided to "ignore the situation completely with unpredictable results". So yes what gcc and clang are doing is hostile and dangerous, but it's legal. https://justine.lol/undefined.png So let's fix our code. The blog post is intended to help people do that.

Re: The Byte Order Fiasco

#244

Post by BeeOnRope »

klodolph wrote: post #171
captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

You don’t have to mask and shift. You can memcpy and then byte swap in a function. It will get inlined as mov/bswap. Practically speaking, common compilers have intrinsics for bswap. The memcpy function can be thought of as an intrinsic for unaligned load/store.

How do you detect if a byte swap is needed? I.e. wether the (fixed) wire endianness matches the current platform endianness?

Re: The Byte Order Fiasco

#245

Post by nwallin »

st_goliath wrote: post #67

FWIW there is a on various BSDs that contains "beXXtoh", "leXXtoh", "htobeXX", "htoleXX" where XX is a number of bits (16, 32, 64). That header is also available on Linux, but glibc (and compatible libraries) named it instead. See: man 3 endian ( https://linux.die.net/man/3/endian ) Of course it gets a bit hairier if the code is also supposed to run on other systems. MacOS has OSSwapHostToLittleIntXX, OSSwapLittleToH…

Windows/MSVC has _byteswap_ushort(), _byteswap_ulong(), _byteswap_uint64(). (note that unsigned long is 32 bits on Windows) It's ugly but it works.

Boost provides boost::endian which allows converting between native and big or little, which just does the right thing on all architectures and compilers and compiles down to a no-op or bswap instruction instruction. It's much better than writing (and testing!) your own giant pile macros and ifdefs to detect the compiler/architecture/OS, include the correct includes, and perform the correct conversions in the correct places.

Re: The Byte Order Fiasco

#246

Post by mitchs »

Or just cast the pointer to uint##_t and use be##toh and htobe## from ? I think this is making a mountain out of a mole hill. I've spent tons of time doing wire (de)serialization in C for network protocols and endian swaps are far from the most pressing issue I see. The big problem imo is the unsafe practices around buffer handling allowing buffer over runs.

Re: The Byte Order Fiasco

#247

Post by vlovich123 »

raphlinus wrote: post #194
jstimpfle wrote: post #175

Earlier quoted context omitted.

I find you missed the point of the post and the issues described in it. In my estimation, libraries like boost are way too big and way too clever and they create more problems than they solve. Also, they don't make me happy. You're overfocusing on a "problem" that is almost completely irrelevant for most of programming. Big endian is rare to be found (almost no hardware to be found, but some file formats and networki…

I agree with the bulk of this post. Re the anecdata at the end. Have you ever run your code through the sanitizers? I have. CVE-2016-2414 is one of my battle scars, and I consider myself a pretty good programmer who is aware of security implications.

Raph, clearly you're just not as good a programmer as you think you are.

Re: The Byte Order Fiasco

#248

Post by Negitivefrags »

Animats wrote: post #207

Rust gets this right. These primitives are available for all the numeric types. u32::from_le_byte(bytes) // u32 from 4 bytes, little endian u32::from_be_byte(bytes) // u32 from 4 bytes, big endian u32::to_le_bytes(num) // u32 to 4 bytes, little endian u32::to_be_bytes(num) // u32 to 4 bytes, big endian This was very useful to me recently as I had to write the marshaling and un-marshaling for a game networking format…

Unless you are planning on running your game on a mainframe, just don’t bother with endianness for the networking.

Big endian is dead for game developers.

Copy entire arrays of structs onto the wire without fear!

(Just #pragma pack them first)

Re: The Byte Order Fiasco

#249

Post by infradig »

Animats wrote: post #207

Rust gets this right. These primitives are available for all the numeric types. u32::from_le_byte(bytes) // u32 from 4 bytes, little endian u32::from_be_byte(bytes) // u32 from 4 bytes, big endian u32::to_le_bytes(num) // u32 to 4 bytes, little endian u32::to_be_bytes(num) // u32 to 4 bytes, big endian This was very useful to me recently as I had to write the marshaling and un-marshaling for a game networking format…

There are equivalent functions in C too. The point of the article is about not using them. So how would you implement the above functions in Rust would be more pertinent.

Re: The Byte Order Fiasco

#250

Post by edflsafoiewq »

BeeOnRope wrote: post #244
klodolph wrote: post #171

Earlier quoted context omitted.

You don’t have to mask and shift. You can memcpy and then byte swap in a function. It will get inlined as mov/bswap. Practically speaking, common compilers have intrinsics for bswap. The memcpy function can be thought of as an intrinsic for unaligned load/store.

How do you detect if a byte swap is needed? I.e. wether the (fixed) wire endianness matches the current platform endianness?

Ie how do you know the target's endianness? C++20 added std::endian. Otherwise you can use a macro like this one from SDL

https://github.com/libsdl-org/SDL/blob/9dc97afa7190aca5bdf92...

Post reply on HN