Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

361–370 of 378 posts

Re: The Byte Order Fiasco

#361

Post by user-the-name »

account42 wrote: post #359
user-the-name wrote: post #355

Earlier quoted context omitted.

Or, you can just follow the advice of the article, and not need to worry about it because the compiler takes care of it for you.

> because the compiler takes care of it for you. It will always be correct, but you can't just assume that the compiler will optimize the shifts into a byteswap instructions. If you look at the article you will see that it tires to no-true-scotsman that concern away by talking about a "good modern compiler".

And what exactly is the problem there? Are you going to be writing code that a) is built with a weird enough compiler that it fails this optimisation but also b) does byte swapping in a performance critical section?

Re: The Byte Order Fiasco

#362

Post by account42 »

jart wrote: post #85
pabs3 wrote: post #17

Earlier quoted context omitted.

IBM is going to be pretty annoyed when your code doesn't work on their mainframes.

In my experience IBM does the right thing and sends patches rather than asking us to fix their problems for them, and I respect them for that reason, even if it's a tiny burden to review those changes. However endianness isn't just about supporting IBM. Modern compilers will literally break your code if you alias memory using a type wider than char. It's illegal per the standard. In the past compilers would simply no…

> It's also worth noting that people think x86 architecture permits unaligned reads but that's not entirely true. For example, you can't do unaligned read-ahead on C strings, because in extremely rare cases you might cross a page boundary that isn't defined and trigger a segfault.

But that's not a problem with an unaligned read but rather that you are reading more than you are allowed to. And in C even an aligned readahead is UB.

A better example might be SSE instructions which do have aligned variants that trap on unaligned pointers.

Re: The Byte Order Fiasco

#363

Post by spacechild1 »

BlueTemplar wrote: post #356
spacechild1 wrote: post #330

Earlier quoted context omitted.

But Unicode itself doesn't! Anyway, it doesn't make much sense to define the size of a “byte“ as anything else then 8 bits, because that's the smallest adressable memory unit. If you need a 32 bit data type, just use one!

My very point is that we should have increased the size of the smallest addressable memory unit from 8 to 32 bits, increased again , as previous computer architectures used from 4 to 7 bits per byte. (There might be still e-mail servers around directly compatible with "non-padded" 7-bit ASCII ?)

But why? Just so that we need to do more bit twiddling and waste memory?

Again, bytes are not foremost about text. We habe to deal with all sorts of data, many of which is shorter than 32 bits.

You can always pick a larger data type for your type of work, but not the opposite.

Re: The Byte Order Fiasco

#364

Post by jart »

BlueTemplar wrote: post #357
jart wrote: post #347

Earlier quoted context omitted.

We're debating semantics, but if I reshaped an RGB image into component arrays i.e. u8[yn][xn][3] → u8[3][yn][xn] then would you still view that as a 24-bit format? What if those 24-bit values were huffman or run-length encoded would it be an n-bit format? If your Y′CbCr luminance plane has a legal range of 16..235 and the chrominance planes are 16..240, then would it be a 23.40892 bit format?

I'm arguing about non-compressed, eventually padded data types that make learning Unicode (or any other applicable data format) easier because of the equivalence : 1 atomic unit ("character", pixel) = 1 smallest addressable unit of memory (byte). This involves byte size being at least as large as atom size. And it's particularly important to have this property for text, because not only data is overwhelmingly stored…

Can you recommend me a good PC computer display at any cost that has an objectively good gamut so I can see what you see?

Re: The Byte Order Fiasco

#365

Post by BlueTemplar »

spacechild1 wrote: post #363
BlueTemplar wrote: post #356

Earlier quoted context omitted.

My very point is that we should have increased the size of the smallest addressable memory unit from 8 to 32 bits, increased again , as previous computer architectures used from 4 to 7 bits per byte. (There might be still e-mail servers around directly compatible with "non-padded" 7-bit ASCII ?)

But why? Just so that we need to do more bit twiddling and waste memory? Again, bytes are not foremost about text. We habe to deal with all sorts of data, many of which is shorter than 32 bits. You can always pick a larger data type for your type of work, but not the opposite.

Because these days it's critical for "basic computer literacy" :

https://news.ycombinator.com/item?id=27094663

https://news.ycombinator.com/item?id=27104860

(You'll also notice that caring about not wasting the 8th bit with ASCII has lead us into all sorts of issues... and why care so much about it when as soon as data density becomes important, we can use compression which AFAIK easily rids us of padding ?)

Re: The Byte Order Fiasco

#366

Post by BlueTemplar »

jart wrote: post #364
BlueTemplar wrote: post #357

Earlier quoted context omitted.

I'm arguing about non-compressed, eventually padded data types that make learning Unicode (or any other applicable data format) easier because of the equivalence : 1 atomic unit ("character", pixel) = 1 smallest addressable unit of memory (byte). This involves byte size being at least as large as atom size. And it's particularly important to have this property for text, because not only data is overwhelmingly stored…

Can you recommend me a good PC computer display at any cost that has an objectively good gamut so I can see what you see?

I'm sorry, I'm not sure that I understand ?

Re: The Byte Order Fiasco

#367

Post by spacechild1 »

BlueTemplar wrote: post #365
spacechild1 wrote: post #363

Earlier quoted context omitted.

But why? Just so that we need to do more bit twiddling and waste memory? Again, bytes are not foremost about text. We habe to deal with all sorts of data, many of which is shorter than 32 bits. You can always pick a larger data type for your type of work, but not the opposite.

Because these days it's critical for "basic computer literacy" : https://news.ycombinator.com/item?id=27094663 https://news.ycombinator.com/item?id=27104860 (You'll also notice that caring about not wasting the 8th bit with ASCII has lead us into all sorts of issues... and why care so much about it when as soon as data density becomes important, we can use compression which AFAIK easily rids us of padding ?)

You're basically arguing against variable width text encodings - which is ok. But you know, it's entirely possible to use UTF32. In fact, some programming languages use it by default to represent strings.

But again and again, all of this has nothing to do with the size of a byte.

BTW, are you aware that 8-bit Microcontrollers are still in widespread use and nowhere near of being discontinued?

Re: The Byte Order Fiasco

#368

Post by jart »

saagarjha wrote: post #275
tsimionescu wrote: post #267

Earlier quoted context omitted.

It should be perfectly fine to do this: union reinterpret { char raw[100]; struct myStruct interpreted; } example; read(fd, &example.raw) struct myStruct dest = interpreted; This is standard-compliant C code, and it is a common way of reading IP addresses from packets, for example.

(It should be noted that this is not valid C++ code.)

Sensitive emotional subjects shouldn't be noted. Reminding C developers of the void* incompatibility is a good way to get them to feel triggered because it makes the language unpleasant.

Re: The Byte Order Fiasco

#369

Post by jart »

BlueTemplar wrote: post #356
spacechild1 wrote: post #330

Earlier quoted context omitted.

But Unicode itself doesn't! Anyway, it doesn't make much sense to define the size of a “byte“ as anything else then 8 bits, because that's the smallest adressable memory unit. If you need a 32 bit data type, just use one!

My very point is that we should have increased the size of the smallest addressable memory unit from 8 to 32 bits, increased again , as previous computer architectures used from 4 to 7 bits per byte. (There might be still e-mail servers around directly compatible with "non-padded" 7-bit ASCII ?)

https://datatracker.ietf.org/doc/html/rfc4042

Re: The Byte Order Fiasco

#370

Post by jart »

spacechild1 wrote: post #367
BlueTemplar wrote: post #365

Earlier quoted context omitted.

Because these days it's critical for "basic computer literacy" : https://news.ycombinator.com/item?id=27094663 https://news.ycombinator.com/item?id=27104860 (You'll also notice that caring about not wasting the 8th bit with ASCII has lead us into all sorts of issues... and why care so much about it when as soon as data density becomes important, we can use compression which AFAIK easily rids us of padding ?)

You're basically arguing against variable width text encodings - which is ok. But you know, it's entirely possible to use UTF32. In fact, some programming languages use it by default to represent strings. But again and again, all of this has nothing to do with the size of a byte. BTW, are you aware that 8-bit Microcontrollers are still in widespread use and nowhere near of being discontinued?

UTF32 is a variable length encoding if we consider combining characters.
Post reply on HN