Live data from Hacker News

The Byte Order Fiasco

justine.lol

201–210 of 378 posts

Re: The Byte Order Fiasco

#201
post #152

Why mask and then shift instead of casting to the correct type and then shifting, like this: (uint32_t)x[0] Of course, this requires that x[0] be unsigned.

If this is for deserialisation then it's okay for x[0] to be signed. You just need to recast the result as int32_t (or simply assign to an int32_t variable without any cast) and it is not UB.

Re: The Byte Order Fiasco

#202
post #23

This is why, in 2021, the mantra that C is a good language for these low level byte twiddling tasks needs to die. Dealing with alignment and endianness properly requires a language that allows you to build abstractions. The following is perfectly well defined in C++, despite looking like almost the same as the original unsafe C: #include #include using namespace boost::endian; unsigned char b[5] = {0x80,0x01,0x02,0x0…

I find you missed the point of the post and the issues described in it. In my estimation, libraries like boost are way too big and way too clever and they create more problems than they solve. Also, they don't make me happy. You're overfocusing on a "problem" that is almost completely irrelevant for most of programming. Big endian is rare to be found (almost no hardware to be found, but some file formats and networki…

Do you use UBSAN and ASAN? When you write unit tests do you feed numbers like 0x80000000 into your algorithm? When you allocate test memory have you considered doing it with mmap(4096) and putting the data at the end of the map? (Or better yet, double it and use mprotect). Those are some good examples of torture tests if you're in the mood to feel haunted.

Re: The Byte Order Fiasco

#203
post #145

Earlier quoted context omitted.

Sorry for being unclear. My point is that the example in the blog post does two things, a) it reads an unaligned address causing UB and b) it performs byte-order swapping. The post then goes on about avoiding UB in part b), but all the time the UB was caused by the unaligned access in a). Of course your example solves both a) and b) by using big_uint32_t, and I agree that this is an interesting abstraction provided b…

Sure, the only correct way to read an unaligned value in to an aligned data type in both C or C++ is via memcpy. I still think being able to define a type that models what you're doing is incredibly valuable because as long as you don't step outside your type system you get so much for free.

You could also mask and shift the value byte-wise just like with an endian swap. Depending on the destination and how aggressive the compiler optimizes memcpy or not, it could even produce more optimal code, perhaps by working in registers more.

Conceptual consistency is a good thing, but there is a generally higher cognitive load to using C++ over C. I've used both C++ and C professionally, and I've gone deeper with type safety and metaprogramming than most folk. I've mostly used C for the last few years, and I don't feel like I'm missing anything. It's still possible to write hard-to-misuse code by coming up with abstractions that play to the language's strengths.

Operator overloading in particular is something I've refined my opinion on over the years. My current thought is that it's best not to use operators in user/application defined APIs, and should be reserved for implementing language defined "standard" APIs like the STL. Instead, it's better to use functions with names that unambiguously describe their purpose.

Re: The Byte Order Fiasco

#204
post #64

https://twitter.com/m13253/status/1371615680068526081 Would it hurt anyone to define this undefined behavior and do exactly what the source code says?

Not sure what you think the source code "says". I mean, I know what you want it to mean, but just because integer wrapping is intuitive to you doesn't imply that that is what the code means. C++ abstract machine and all. But to answer the actual question: For C++20, integer types were revisited. It is now (finally) guaranteed that signed integers are two's complement, along with a list of other changes. See http://ww…

I haven't noticed the signed integer overflow, which does indeed complicate things, and I thought it was just the infinite loop UB.

> Data from Google suggesting that over 90% of all overflow is a bug, and defining wrapping behavior would not have solved the bug.

Of all overflow? Including unsigned integers where the behavior is defined?

Re: The Byte Order Fiasco

#205

FWIW there is a on various BSDs that contains "beXXtoh", "leXXtoh", "htobeXX", "htoleXX" where XX is a number of bits (16, 32, 64). That header is also available on Linux, but glibc (and compatible libraries) named it instead. See: man 3 endian ( https://linux.die.net/man/3/endian ) Of course it gets a bit hairier if the code is also supposed to run on other systems. MacOS has OSSwapHostToLittleIntXX, OSSwapLittleToH…

Indeed, I don't get the article. It's like writing "C is hard because here is how hard it is to implement memcpy using SIMD correctly."

Please don't do that. Use battle-tested low-level routines. Unless your USP is "our software swaps bytes faster than the competition", you should not spend brain power on that.

Re: The Byte Order Fiasco

#206

In an ideal world which endian format would one go for?

I for one would go for big-endian, simply because reading memory dumps and byte blocks in assembly or elsewhere works without mental byte-swapping arithmetics for multi-byte entities. Just out of curiosity, I would be interested in learning why so many CPUs today are little-endian. Is it because it is cheaper / more efficient for processor implementations or is it because “the others do it, so we do it the same way”?

My brain is trained to read little-endian in memory dumps. It's no different than the German "fünf-und-zwanzig" (five and twenty). :))

Re: The Byte Order Fiasco

#207
Rust gets this right. These primitives are available for all the numeric types.

    u32::from_le_byte(bytes) // u32 from 4 bytes, little endian
    u32::from_be_byte(bytes) // u32 from 4 bytes, big endian
    u32::to_le_bytes(num) // u32 to 4 bytes, little endian
    u32::to_be_bytes(num) // u32 to 4 bytes, big endian
This was very useful to me recently as I had to write the marshaling and un-marshaling for a game networking format with hundreds of messages. With primitives like this, you can see what's going on.

Re: The Byte Order Fiasco

#208
post #26

Earlier quoted context omitted.

That reminds me of a project to interface with vending machines. (We built a bookshop in a vending machine that would tweet whenever it sold an item, with automated stock management.) Vending machines have an internal protocol a little like I2C. We created a custom peripheral to bridge the machine to the web, based on a Raspberry Pi. The protocol was defined by Coca Cola Japan in 1975 (in order to have optionality in…

This just doesn't seem right. Granted, I don't know much about your use case, but Raspberry Pi's are powerful computing devices and I find it difficult to believe there was no way to handle this without additional hardware.

I’m not familiar with the “vending machine” protocol he’s talking about, but it’s entirely reasonable that it has certain timing requirements. Usually the way you interface with these is by having a dedicated HW block to talk the protocol, or by bit banging. The former wouldn’t be supported on RPi because it’s obscure, the latter requires tight GPIO timing control that is difficult to guarantee on a non-real-time system like the RPi usually runs.

Re: The Byte Order Fiasco

#209

Earlier quoted context omitted.

I find you missed the point of the post and the issues described in it. In my estimation, libraries like boost are way too big and way too clever and they create more problems than they solve. Also, they don't make me happy. You're overfocusing on a "problem" that is almost completely irrelevant for most of programming. Big endian is rare to be found (almost no hardware to be found, but some file formats and networki…

I agree with the bulk of this post. Re the anecdata at the end. Have you ever run your code through the sanitizers? I have. CVE-2016-2414 is one of my battle scars, and I consider myself a pretty good programmer who is aware of security implications.

[deleted]

Re: The Byte Order Fiasco

#210
post #178
post #173

In her first sentence, the phrase “the C / C++ programming language” is no longer correct: C++20 requires two’s complement signed integers. C++ 20 is quite new so I would assume that very few people know this yet. C and C++ obviously differ a lot, but by that phrase she clearly means “the part where then two languages overlap”. The C++ committee has been willing to break C compatibility in a few ways (not every valid…

What chips can be targeted by C compilers today that don't use 2's complement?

I haven’t seen a one’s complement machine in decades but at the time C was standardized here were still quite a few (afaik none had a single-chip CPU, to get to your question). But since they existed, the language definition didn’t require it and some optimizations were technically UB.

The C++ committee decided that everyone had figured this out by now and so made this breaking change.

Post reply on HN