Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

141–150 of 378 posts

Re: The Byte Order Fiasco

#141

Post by tails4e »

MauranKilom wrote: post #134
tails4e wrote: post #6

Ubsan should default on. If people don't like it, then they should be made turn it off with a switch, so at least it's more likely to be run than not run. Could save a huge amount of time debugging when compilers or architecture changes. Without it, I'd say many a programmer would be caught by these subtleties in the standard. Coming from a HW background (Verilog) I'd more naturally default to masking and shifting wh…

> Ubsan should default on > Could save a huge amount of time debugging when compilers or architecture changes. I'm assuming we come from very different backgrounds, but it's not clear to me how switching compilers or architectures is so common that hardening code against it by default is appropriate. I would think that switching compilers or architectures is generally done very deliberately, so instrumenting code wit…

Changing gcc version could cause your code with undefined behaviour to change. If you rely UB, whether you know you are or not, you are in for a bad time. Ubsan at least let's you know if your code is robust, or a ticking time bomb...

Re: The Byte Order Fiasco

#142

Post by thristian »

ghoward wrote: post #105
raverbashing wrote: post #32

Earlier quoted context omitted.

Well you could bit bang and the 9 bits wouldn't be an issue. (Even if you had a tiny PIC microcontroler just to do that) This is best solvable the closer to the device in question and in the simplest way possible.

Sorry, dumb question: what is bit banging?

In order to exchange data over a serial connection, the ones and zeroes have to be sent with exact timing, so the receiver can reliably tell where one bit ends and the next begins. Because of this, the hardware that's doing the communication can't do anything else at the same time. And since the actual mechanics of the process are simple and straightforward, most computers with a serial connection have special serial-interface hardware (a Universal Asynchronous Receiver/Transmitter, or UART) to take care of it - the CPU gives the UART some data, then returns to more productive pursuits while the UART works away.

But sometimes you can't use a UART: maybe you're working on a tiny embedded computer without one, or maybe you need to speak a weird 9-bit protocol a standard UART doesn't understand. In that case, you can make the CPU pump the serial line directly. It's inefficient (there's probably more interesting work the CPU could be doing) and it can be difficult to make the CPU pause for exactly the right amount of time (CPUs are normally designed to run as fast or as efficiently as possible, nothing in between), but it's possible and sometimes it's all you've got. That's bit-banging.

Re: The Byte Order Fiasco

#143

Post by MauranKilom »

ttt0 wrote: post #64

https://twitter.com/m13253/status/1371615680068526081 Would it hurt anyone to define this undefined behavior and do exactly what the source code says?

Not sure what you think the source code "says". I mean, I know what you want it to mean, but just because integer wrapping is intuitive to you doesn't imply that that is what the code means. C++ abstract machine and all.

But to answer the actual question: For C++20, integer types were revisited. It is now (finally) guaranteed that signed integers are two's complement, along with a list of other changes. See http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p090... also for how the committee voted on the individual issues.

Note in particular:

> The main change between [P0907r0] and the subsequent revision is to maintain undefined behavior when signed integer overflow occurs, instead of defining wrapping behavior. This direction was motivated by:

> - Performance concerns, whereby defining the behavior prevents optimizers from assuming that overflow never occurs;

> - Implementation leeway for tools such as sanitizers;

> - Data from Google suggesting that over 90% of all overflow is a bug, and defining wrapping behavior would not have solved the bug.

So yes, the committee very recently revisited this specific issue, and re-affirmed that signed integer overflow should be UB.

Re: The Byte Order Fiasco

#144

Post by jart »

cygx wrote: post #107
BlueTemplar wrote: post #90

Earlier quoted context omitted.

One byte = one "character" makes for much easier programming. Text generally uses a small fraction of memory and storage these days.

Not all user-perceived characters can be represented as a single Unicode codepoint. Hence, Unicode text encodings (almost[1]) always have to be treated as variable length, even UTF-32. [1] at runtime, you could dynamically assign 'virtual' codepoints to grapheme clusters and get a fixed-length encoding for strings that way

Even the individual unicode codepoints themselves are variable width if we consider that things like cjk and emoji take up >1 monospace cells.

Re: The Byte Order Fiasco

#145

Post by nly »

themulticaster wrote: post #130
nly wrote: post #112

Earlier quoted context omitted.

I fail to see your point. The point of my post is that the abstractions you can build in C++ are as easy to use and as efficient as doing things the wrong, unsafe way...so there's no reason not to do things in a safe, correct way. Obviously if you write C and compile it as C++ you still end up with UB, because C++ aims for extreme levels of compatibility with C.

Sorry for being unclear. My point is that the example in the blog post does two things, a) it reads an unaligned address causing UB and b) it performs byte-order swapping. The post then goes on about avoiding UB in part b), but all the time the UB was caused by the unaligned access in a). Of course your example solves both a) and b) by using big_uint32_t, and I agree that this is an interesting abstraction provided b…

Sure, the only correct way to read an unaligned value in to an aligned data type in both C or C++ is via memcpy.

I still think being able to define a type that models what you're doing is incredibly valuable because as long as you don't step outside your type system you get so much for free.

Re: The Byte Order Fiasco

#146

Post by raphlinus »

nly wrote: post #120
raphlinus wrote: post #99

Earlier quoted context omitted.

The idiomatic way to do this in Rust is to use functions like .to_le_bytes(), so you have the u32 (or whatever) on one end and raw bytes (something like [u8; 4]) on the other. It can get slightly tedious if you're doing it by hand, but it's impossible to accidentally forget. If you're doing this kind of thing at scale, like dealing with TrueType fonts (another bastion of big-endian), it's common to reach for derive m…

Who decides what methods to add to the bytes type/abstraction? If I have a 3 byte big endian integer can I access it easily in rust without resorting to shifts? In C++ I could probably create a fairly convincing big_uint24_t type and use it in a packed struct and there would be no inconsistencies with how it's used with respect to the more common varieties

In Rust, [u8; N] and &[u8] are both primitive types, and not abstractions. It's possible to create an abstraction around either (the former even more so now with const generics), but that's not necessary. It's also possible to use "extension traits" to add methods, even to existing and built-in types[1].

I'm not sure about a 3 byte big endian integer. I mean, that's going to compile down to some combination of shifting and masking operations anyway, isn't it? I suspect that if you have some oddball binary format that needs, this it will be possible to write some code to marshal it, that compiles down to the best possible asm. Godbolt is your friend here :)

[1]: https://rust-lang.github.io/rfcs/0445-extension-trait-conven...

Re: The Byte Order Fiasco

#147

Post by joppy »

amelius wrote: post #3

Byte order is one of the great unnecessary historical fuck ups in computing. A similar one is that signedness of char is machine dependent. It's typically signed on Intel and unsigned on ARM. Sigh!

Why is it an issue any more than say, order of fields in a struct is an issue? In one case you read bytes off the disk by doing ((b[0] << 8) | b[1]) (or equivalent), with the order reversed the other way around. Any application-level (say, not a compiler, debugger, etc) program should not even need to know the native byte order, it should only need to know the encoding that the file it’s trying to read used.

Re: The Byte Order Fiasco

#148

Post by mfost »

foldr wrote: post #101
nly wrote: post #23

This is why, in 2021, the mantra that C is a good language for these low level byte twiddling tasks needs to die. Dealing with alignment and endianness properly requires a language that allows you to build abstractions. The following is perfectly well defined in C++, despite looking like almost the same as the original unsafe C: #include #include using namespace boost::endian; unsigned char b[5] = {0x80,0x01,0x02,0x0…

What are the advantages of this over a simple function with the following signature? uint32_t read_big_uint32(char *bytes); Having a big_uint32_t type seems wrong to me conceptually. You should either deal with sequences of bytes with a defined endianness or with native 32-bit integers of indeterminate endianness (assuming that your code is intended to be endian neutral). Having some kind of halfway house just confus…

I'd say, putting multiple of those types into a struct that then perfectly describes the memory layout of each byte of data in memory/network packet in a reliable and user friendly way to manipulate for the coder.

Re: The Byte Order Fiasco

#149

Post by kingsuper20 »

I've never been very satisfied with these approaches for C where you hope the compiler does the right thing. It makes sense to provide some C implementation for portability's sake but any sizeable reordering cries out for a handtuned, processor specific, approach (and the non-sizeable probably doesn't require high speed). I would expect any SIMD instruction set to include a shuffle.
Post reply on HN