Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

321–330 of 378 posts

Re: The Byte Order Fiasco

#321

Post by BlueTemplar »

jart wrote: post #122
BlueTemplar wrote: post #95

Earlier quoted context omitted.

Well, obviously it would have delayed the transition. However you can only go so far with 4Go-limited memory. And do you have examples of still widely used 8-bit sized data formats ?

RGB and Y′CbCr

To start with, RGB (and I assume Y′CbCr ?) can be encoded in many different ways. The most common one today (still) uses 8 bits per channel, meaning that a separated 1-octet value can only define monochrome. Therefore 8-bpc RGB is a 24-bit sized format, not a 8-bit sized data one.

And, by an interesting coincidence, with the arrival of "HDR", 8-bit per channel is slowly becoming obsolete (because insufficient). The next "step" is 10-bit per channel with 3 channels (hence "HDR10(+)"), and so should fit quite well in 32 bits ?

(However, it would seem that even Dolby's Perceptual Quantizer transfer function might need 12 bits per channel to avoid banding over the "HDR" Rec.2020/2100-sized color gamut..?)

Re: The Byte Order Fiasco

#322

Post by secondcoming »

xeeeeeeeeeeenu wrote: post #299
secondcoming wrote: post #69

Isn't the 'modern' solution to memcpy into a temp and swap the bytes in that? C++ has added/will add std::launder and std::bless to deal with this issue

>C++ has added/will add std::launder and std::bless to deal with this issue You're thinking of std::bit_cast. std::launder solves a different, much more obscure problem: https://miyuki.github.io/2016/10/21/std-launder.html

oops, my mistake

Re: The Byte Order Fiasco

#323

Post by BlueTemplar »

kortex wrote: post #154
BlueTemplar wrote: post #90

Earlier quoted context omitted.

One byte = one "character" makes for much easier programming. Text generally uses a small fraction of memory and storage these days.

> One byte = one "character" makes for much easier programming. Only if you are naively operating in the Anglosphere / world where the most complex thing you have to handle is larger character sets. In reality, there's ligatures, diacritics, combining characters, RTL, nbsp, locales, and emoji (with skin tones!). Not to mention legacy encoding. And no, it does not use a "small fraction of memory and storage" in a huge…

This is not about covering ALL of Unicode. This is about starting to cover Unicode.

"Anglosphere" would be just 7(&"8") bit ASCII, and it's the current situation where it takes quite a lot of skill and knowledge just to start learning how to properly deal with Unicode, because it's often not even taught !

IMHO 32-bit bytes would help tremendously with onboarding developers into Unicode, because it would force dumping ASCII-only as the starting point (and sadly, often ending point) for teaching how to deal with text.

And who can blame the teachers, Unicode is already hard enough without even having to deal with the difficulties coming from having to explain its multi-byte representation...

Last but not least : this would have forced standardization between the Unix world now on UTF-8 and the Windows world which is still stuck on UTF-16 (and Windows-1252 ?!?) for some of the core functions like filenames, which, for instance, still regularly results in issues working with files with non-ASCII filenames.

Re: The Byte Order Fiasco

#324

Post by BlueTemplar »

spacechild1 wrote: post #119
BlueTemplar wrote: post #36

Earlier quoted context omitted.

We really should have moved to 32 bit bytes when moving to 64 bit words. Would have simplified Unicode considerably.

You know, bytes are not only about text, they are also used to represent binary data... Not to mention that bytes have nothing to do with unicode. Unicode codepoints can be encoded in many different ways: UTF8, UTF16, UTF32, etc.

https://news.ycombinator.com/item?id=27086928

These various ways to encode Unicode have quite a lot to do with bytes being 8-bit sized !

Re: The Byte Order Fiasco

#325

Post by user-the-name »

klodolph wrote: post #302
user-the-name wrote: post #240

Earlier quoted context omitted.

When do you byte swap?

24 hours a day, man. I'm always byte swapping. (I'm not sure how to answer the question... what do you mean, "when?")

The entire problem of using byte swaps is that you need to use them when your native platform's byte order is different from that of the data you are reading.

You know the byte order of the data. But the tricky part is, what is the byte order of the platform?

Re: The Byte Order Fiasco

#326

Post by Measter »

infradig wrote: post #249
Animats wrote: post #207

Rust gets this right. These primitives are available for all the numeric types. u32::from_le_byte(bytes) // u32 from 4 bytes, little endian u32::from_be_byte(bytes) // u32 from 4 bytes, big endian u32::to_le_bytes(num) // u32 to 4 bytes, little endian u32::to_be_bytes(num) // u32 to 4 bytes, big endian This was very useful to me recently as I had to write the marshaling and un-marshaling for a game networking format…

There are equivalent functions in C too. The point of the article is about not using them. So how would you implement the above functions in Rust would be more pertinent.

If I were forced to implement them myself for some reason, I would probably simply do them like this:

    fn from_be(bytes: [u8; 4]) -> u32 {
        (bytes[0] as u32)
It's direct, to the point, and does exactly what it says on the tin because all pertinent behaviour is defined. The way Rust's corelib implements it is to transmute the array into the integer, then call the bswap intrinsic if the bytes need swapping(detected at compile time).

Re: The Byte Order Fiasco

#327

Post by tempodox »

mrlonglong wrote: post #165
bombcar wrote: post #96

Earlier quoted context omitted.

https://stackoverflow.com/questions/5185551/why-is-x86-littl... It simplifies certain instructions internally. Practically everything is little endian because x86 won. > And if you think about a serial machine, you have to process all the addresses and data one-bit at a time, and the rational way to do that is: low-bit to high-bit because that’s the way that carry would propagate. So it means that [in] the jump instr…

And does middle endian even exist?

Not currently AFAIK, but apparently the PDP-11 had a middle-endian arch. See other comments in this thread.

Re: The Byte Order Fiasco

#328

Post by loup-vaillant »

Here's how I implement little endian parsing:

  static uint32_t load32_le(const uint8_t s[4])
  {
      return (uint32_t)s[0]
          | ((uint32_t)s[1]
I start with unsigned char to begin with (well `uint8_t` to be precise, which has the advantage of not compiling at all if you happen to use a DSP that uses 32-bit chars). Then I convert those chars to unsigned 32-bit integers. Only then do I shift them. There is no need to mask anything here.

Note that modern compilers translate this whole thing into a single unaligned load operation. Even better, I've noticed that using a macro instead of a function tends to make performance worse with modern compilers.

Re: The Byte Order Fiasco

#329

Post by tails4e »

mytherin wrote: post #319
tails4e wrote: post #274

Earlier quoted context omitted.

Sorry for my ignorance, but surely some UB being used for optimization by the compiler is compile time only. This is the part that should default on. Runtime detection is a different thing entirely, but compile time is a no brainer.

UBSAN detects undefined behavior at run-time. Compile-time detection of undefined behavior is present in the form of compiler warnings, but catches far from all cases of undefined behavior. The compiler does not actively exploit undefined behavior in the sense that it does not contain code like this: if (undefined_behavior) break_program() If it did, it could easily report the undefined behavior. However, that's not…

Thanks, though I'm not sure all compile time detectable undefined behaviours is exposed through warnings today. In the example in the article, why would something like left shift of a -ve value require runtime detection, surely the fact a signed char was used with left shift is all the compiler needs. So perhaps a subset to UB that is detectable at compile time should be reported.

In your example about the comparison of x + 1 vs x, I'm not sure that is a contraversial optimization. However this one, to me, is:

http://blog.llvm.org/2011/05/what-every-c-programmer-should-...

Here a diligent programmer is trying to do a null pointer check, but because dereferencing null is UB, then the optimizer removes the null pointer check. This is compile time UB that should be flagged to users.

Re: The Byte Order Fiasco

#330

Post by spacechild1 »

BlueTemplar wrote: post #324
spacechild1 wrote: post #119

Earlier quoted context omitted.

You know, bytes are not only about text, they are also used to represent binary data... Not to mention that bytes have nothing to do with unicode. Unicode codepoints can be encoded in many different ways: UTF8, UTF16, UTF32, etc.

https://news.ycombinator.com/item?id=27086928 These various ways to encode Unicode have quite a lot to do with bytes being 8-bit sized !

But Unicode itself doesn't!

Anyway, it doesn't make much sense to define the size of a “byte“ as anything else then 8 bits, because that's the smallest adressable memory unit. If you need a 32 bit data type, just use one!

Post reply on HN