Live data from Hacker News

The Byte Order Fiasco

justine.lol

111–120 of 378 posts

Re: The Byte Order Fiasco

#111
post #6

Ubsan should default on. If people don't like it, then they should be made turn it off with a switch, so at least it's more likely to be run than not run. Could save a huge amount of time debugging when compilers or architecture changes. Without it, I'd say many a programmer would be caught by these subtleties in the standard. Coming from a HW background (Verilog) I'd more naturally default to masking and shifting wh…

Sanitizers may introduce side channels. This is an issue for crypto code.

Re: The Byte Order Fiasco

#112
post #23

This is why, in 2021, the mantra that C is a good language for these low level byte twiddling tasks needs to die. Dealing with alignment and endianness properly requires a language that allows you to build abstractions. The following is perfectly well defined in C++, despite looking like almost the same as the original unsafe C: #include #include using namespace boost::endian; unsigned char b[5] = {0x80,0x01,0x02,0x0…

This has nothing to do with C++ because your example only hides the real issue occurring in the blog post example: The unaligned read on the array. Try adding something like printf("%08x\n", *((uint32_t*)(b))); to your example and you'll see that it produces UB as well. The reason there is no UB with big_uint32_t probably is that that struct/class/whatever it is probably redefines its dereferencing operator to perfor…

I fail to see your point. The point of my post is that the abstractions you can build in C++ are as easy to use and as efficient as doing things the wrong, unsafe way...so there's no reason not to do things in a safe, correct way.

Obviously if you write C and compile it as C++ you still end up with UB, because C++ aims for extreme levels of compatibility with C.

Re: The Byte Order Fiasco

#113
post #26

Remember how we used to have machines with a 7 bit byte? And everything was written to handle either 6, 7, or 8 bit bytes? And now we've settled on all machines being 8 bit bytes, and programmers no longer have to worry about such details? Is it time to do the same for big endian machines? Is it time to accept that all machines that matter are little endian, and the extra effort keeping everything portable to big end…

That reminds me of a project to interface with vending machines. (We built a bookshop in a vending machine that would tweet whenever it sold an item, with automated stock management.) Vending machines have an internal protocol a little like I2C. We created a custom peripheral to bridge the machine to the web, based on a Raspberry Pi. The protocol was defined by Coca Cola Japan in 1975 (in order to have optionality in…

This just doesn't seem right. Granted, I don't know much about your use case, but Raspberry Pi's are powerful computing devices and I find it difficult to believe there was no way to handle this without additional hardware.

Re: The Byte Order Fiasco

#114
post #101
post #23

This is why, in 2021, the mantra that C is a good language for these low level byte twiddling tasks needs to die. Dealing with alignment and endianness properly requires a language that allows you to build abstractions. The following is perfectly well defined in C++, despite looking like almost the same as the original unsafe C: #include #include using namespace boost::endian; unsigned char b[5] = {0x80,0x01,0x02,0x0…

What are the advantages of this over a simple function with the following signature? uint32_t read_big_uint32(char *bytes); Having a big_uint32_t type seems wrong to me conceptually. You should either deal with sequences of bytes with a defined endianness or with native 32-bit integers of indeterminate endianness (assuming that your code is intended to be endian neutral). Having some kind of halfway house just confus…

The library provides those functions too, but I don't see how having an arithmetic type with well defined size, endiannness and alignment is a bad thing.

If you're defining a struct to mirror a data structure from a device, protocol or file format then the language / type system should let you define the properties of the fields, not necessarily force you to introduce a parsing/decoding stage which could be more easily bypassed.

Re: The Byte Order Fiasco

#115
post #26

Earlier quoted context omitted.

That reminds me of a project to interface with vending machines. (We built a bookshop in a vending machine that would tweet whenever it sold an item, with automated stock management.) Vending machines have an internal protocol a little like I2C. We created a custom peripheral to bridge the machine to the web, based on a Raspberry Pi. The protocol was defined by Coca Cola Japan in 1975 (in order to have optionality in…

We really should have moved to 32 bit bytes when moving to 64 bit words. Would have simplified Unicode considerably.

Use Erlang. It has 32-bit char.

Re: The Byte Order Fiasco

#116

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

It does, macro assemblers, specially those with PC and Amiga roots.

Which given its heritage, that is what PDP-11 C used to be, after all BCPL origin was as minimal language required to bootstrap CPL, nothing else.

Actually, I think TI has a macro Assembler with a C like syntax, just cannot recall the name any longer.

Re: The Byte Order Fiasco

#117

Earlier quoted context omitted.

Exactly. This is so infuriating. Whoever let little-endian win made a huge disfavor for humanity.

The only benefit to big endian is that it's easier for humans to read in a hex dump. Little endian on the other hand has many tricks available to it for building encoding schemes that are efficient on the decoder side.

Could you elaborate on these tricks? This sounds interesting.

The only thing I'm aware of that's neat in little endian is that if you want the low byte (or word or whatever suffix) of a number stored at address a, then you can simply read a byte from exactly that address. Even if you don't know the size of the original number.

Re: The Byte Order Fiasco

#118
post #75

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

> One of the few reasons I ever even reached to C is the ability to slurp in data and reinterpret it as a struct, or the ability to reason in which registers things will show up and mix in some `asm` with my C. Which results in undefined behavior according to the C ISO standard. Quote: “2 All declarations that refer to the same object or function shall have compatible type; otherwise, the behavior is undefined.” From…

Exactly.

Re: The Byte Order Fiasco

#119
post #26

Earlier quoted context omitted.

That reminds me of a project to interface with vending machines. (We built a bookshop in a vending machine that would tweet whenever it sold an item, with automated stock management.) Vending machines have an internal protocol a little like I2C. We created a custom peripheral to bridge the machine to the web, based on a Raspberry Pi. The protocol was defined by Coca Cola Japan in 1975 (in order to have optionality in…

We really should have moved to 32 bit bytes when moving to 64 bit words. Would have simplified Unicode considerably.

You know, bytes are not only about text, they are also used to represent binary data...

Not to mention that bytes have nothing to do with unicode. Unicode codepoints can be encoded in many different ways: UTF8, UTF16, UTF32, etc.

Re: The Byte Order Fiasco

#120
post #79

Earlier quoted context omitted.

That's fine if whatever type 'var' happens to be is NOT usable as an arithmetic type, otherwise you can easily just forget to call .to_le() or .to_native(), or whatever, and end up with a bug. I don't know Rust, so don't know if this is the case? Boost.Endian actually lets you pick between arithmetic and buffer types. 'big_uint32_buf_t' is a buffer type that requires you to call .value() or do a conversion to an inte…

The idiomatic way to do this in Rust is to use functions like .to_le_bytes(), so you have the u32 (or whatever) on one end and raw bytes (something like [u8; 4]) on the other. It can get slightly tedious if you're doing it by hand, but it's impossible to accidentally forget. If you're doing this kind of thing at scale, like dealing with TrueType fonts (another bastion of big-endian), it's common to reach for derive m…

Who decides what methods to add to the bytes type/abstraction?

If I have a 3 byte big endian integer can I access it easily in rust without resorting to shifts?

In C++ I could probably create a fairly convincing big_uint24_t type and use it in a packed struct and there would be no inconsistencies with how it's used with respect to the more common varieties

Post reply on HN