Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

101–110 of 378 posts

Re: The Byte Order Fiasco

#101

Post by foldr »

nly wrote: post #23

This is why, in 2021, the mantra that C is a good language for these low level byte twiddling tasks needs to die. Dealing with alignment and endianness properly requires a language that allows you to build abstractions. The following is perfectly well defined in C++, despite looking like almost the same as the original unsafe C: #include #include using namespace boost::endian; unsigned char b[5] = {0x80,0x01,0x02,0x0…

What are the advantages of this over a simple function with the following signature?

    uint32_t read_big_uint32(char *bytes);
Having a big_uint32_t type seems wrong to me conceptually. You should either deal with sequences of bytes with a defined endianness or with native 32-bit integers of indeterminate endianness (assuming that your code is intended to be endian neutral). Having some kind of halfway house just confuses things.

Re: The Byte Order Fiasco

#102

Post by amelius »

bombcar wrote: post #88
kstenerud wrote: post #43

Earlier quoted context omitted.

Big endian is easier for humans to read when looking at a memory dump, but little endian has many useful features in binary encoding schemes due to the low byte being first. I used to like big endian more, but after deep investigation I now prefer little endian for any encoding schemes.

Couldn’t encoding systems be redone with emphasis on the high-order bits? Or is the assumption that the values are clustered in the low bits?

I think the fundamental problems is that if you start a computation using N most significant bits and then incrementally add more bits, e.g. N+M bits total, then your first N bits might change as a result.

E.g. decimal example:

    1.00/1.00 = 1.00
    1.000/1.001 = 0.999000999000...
(adding one more bit changes the first bits of the outcome)

Re: The Byte Order Fiasco

#103

Post by jart »

LightMachine wrote: post #65
BenoitEssiambre wrote: post #53

Earlier quoted context omitted.

Also, this might be irrelevant at the cpu level, but within a byte, bits are usually displayed most significant bit first, so with little endian you end up with bit order: 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 instead of 15 to 0 This is because little endian is not how humans write numbers. For consistency with little endianness we would have to switch to writing "one hundred and twenty three" as 321

Exactly. This is so infuriating. Whoever let little-endian win made a huge disfavor for humanity.

Blame the people who failed to localize the right-to-left convention when arabic numerals were adopted. It's one of those things like pi vs. tau or jacobin weights and measurements vs. planck units. Tradition isn't always correct. John von Neumann understood that when he designed modern architecture and muh hex dump is not an argument.

Re: The Byte Order Fiasco

#104

Post by [deleted] »

jart wrote: post #63
rwmj wrote: post #38

Earlier quoted context omitted.

Assuming the macros or your giant expression are correct. But you might as well use the compiler intrinsics which you know are both correct and the most efficient possible, and get on with your life.

Sorry I'd rather place my faith in arithmetic rather than someone's API provided the compiler is smart enough to understand the arithmetic and optimize accordingly.

[deleted]

Re: The Byte Order Fiasco

#105

Post by ghoward »

raverbashing wrote: post #32
genmon wrote: post #26

Earlier quoted context omitted.

That reminds me of a project to interface with vending machines. (We built a bookshop in a vending machine that would tweet whenever it sold an item, with automated stock management.) Vending machines have an internal protocol a little like I2C. We created a custom peripheral to bridge the machine to the web, based on a Raspberry Pi. The protocol was defined by Coca Cola Japan in 1975 (in order to have optionality in…

Well you could bit bang and the 9 bits wouldn't be an issue. (Even if you had a tiny PIC microcontroler just to do that) This is best solvable the closer to the device in question and in the simplest way possible.

Sorry, dumb question: what is bit banging?

Re: The Byte Order Fiasco

#106

Post by themulticaster »

cygx wrote: post #91
themulticaster wrote: post #83

The first example in the article is flawed (or at least misleading). 1) They define a char array (which defaults to signed char, as mentioned in the post), including the value 0x80 which can't be represented in char, resulting in a compiler warning (e.g. in GCC 11.1). The mentioned reason against using unsigned char (that shifting 128 left by 24 places results in UB) is also misleading: I could not reproduce the UB w…

> The mentioned reason against using unsigned char (that shifting 128 left by 24 places results in UB) is also misleading No, that reasoning is correct. Integer promotions are performed on the operands of a shift expression, meaning the left operand will be promoted to signed int even if it starts out as unsigned char. Trying to shift a byte value with highest bit set by 24 will results in a value not representable a…

Thanks, I just noticed a small mistake in my example (I don't trigger the UB because I access b[0] containing 0x80 without shifting, however I meant to do it the other way around).

Still, adding an explicit cast to the left operand seems to be enough to avoid this, e.g.:

  uint32_t x = ((uint32_t)b[0])
In summary, I think my point that using unsigned char would be appropriate in this case still stands.

Re: The Byte Order Fiasco

#107

Post by cygx »

BlueTemplar wrote: post #90
ChrisSD wrote: post #48

Earlier quoted context omitted.

Not really. Unicode is a variable width abstract encoding; a single character can be made up of multiple code points. For Unicode, 32-bit bytes would be an incredibly wasteful in memory encoding.

One byte = one "character" makes for much easier programming. Text generally uses a small fraction of memory and storage these days.

Not all user-perceived characters can be represented as a single Unicode codepoint. Hence, Unicode text encodings (almost[1]) always have to be treated as variable length, even UTF-32.

[1] at runtime, you could dynamically assign 'virtual' codepoints to grapheme clusters and get a fixed-length encoding for strings that way

Re: The Byte Order Fiasco

#108

Post by kstenerud »

bombcar wrote: post #88
kstenerud wrote: post #43

Earlier quoted context omitted.

Big endian is easier for humans to read when looking at a memory dump, but little endian has many useful features in binary encoding schemes due to the low byte being first. I used to like big endian more, but after deep investigation I now prefer little endian for any encoding schemes.

Couldn’t encoding systems be redone with emphasis on the high-order bits? Or is the assumption that the values are clustered in the low bits?

You can put emphasis on high order bits, but that makes decoding more complex. With little endian the decoder builds low to high, which is MUCH easier to deal with, especially on spillover.

For example, with ULEB128 [1], you just read 7 bits at a time, going higher and higher up the value you're reconstituting. If the value grows too big and you need to spill over to the next (such as with big integer implementations), you just fill the last bits of the old value, then put the remainder bits in the next value and continue on.

With a big endian encoding method (i.e. VLQ used in MIDI format), you start from the high bits and work your way down, which is fine until your value spills over. Because you only have the high bits decoded at the time of the spillover, you now have to start shifting bits along each of your already decoded big integer portions until you finally decode the lowest bit. This of course gets progressively slower as the bits and your big integer portions pile up.

Encoding is easier too, since you don't need to check if for example a uint64 integer value can be encoded in 1, 2, 3, 4, 5, 6, 7 or 8 bits. Just encode the low 8 bits, shift the source right by 8, repeat, until the source value is 0. Then backtrack to the as-yet-blank encoded length field in your message and stuff in how many bytes you encoded. You just got the length calculation for free. Use a scheme where you only encode up to 60 bit values, place the length field in the low 4 bits, and Robert's your father's brother!

For data that is right-heavy (i.e. the fully formed data always has real data on the right side and blank filler on the left - such as uint32 value 8 is actually 0x00000008), you want a little endian scheme. For data that is left-heavy, you want a big endian scheme. Since most of the data we deal with is right-heavy, little endian is the way to go.

You can see how this has influenced my encoding design in [2] [3] [4].

[1] https://en.wikipedia.org/wiki/LEB128

[2] https://github.com/kstenerud/concise-encoding/blob/master/cb...

[3] https://github.com/kstenerud/compact-float/blob/master/compa...

[4] https://github.com/kstenerud/compact-time/blob/master/compac...

Re: The Byte Order Fiasco

#109

Post by cygx »

themulticaster wrote: post #106
cygx wrote: post #91

Earlier quoted context omitted.

> The mentioned reason against using unsigned char (that shifting 128 left by 24 places results in UB) is also misleading No, that reasoning is correct. Integer promotions are performed on the operands of a shift expression, meaning the left operand will be promoted to signed int even if it starts out as unsigned char. Trying to shift a byte value with highest bit set by 24 will results in a value not representable a…

Thanks, I just noticed a small mistake in my example (I don't trigger the UB because I access b[0] containing 0x80 without shifting, however I meant to do it the other way around). Still, adding an explicit cast to the left operand seems to be enough to avoid this, e.g.: uint32_t x = ((uint32_t)b[0]) In summary, I think my point that using unsigned char would be appropriate in this case still stands.

> Still, adding an explicit cast to the left operand seems to be enough to avoid this

Indeed. See my other comment, https://news.ycombinator.com/item?id=27086482

Post reply on HN