Live data from Hacker News

The Byte Order Fiasco

justine.lol

351–360 of 378 posts

Re: The Byte Order Fiasco

#351

Earlier quoted context omitted.

The address of p could have been taken somewhere earlier and stored in a global that foo accesses, or a similar path to that; and of course, p could itself be a global. Indeed, if the purpose of foo is to make p non-null and point to valid memory, then by optimising away that code you have broken a valid program. If the compiler doesn't know if foo may modify p, then it can't remove the call. Even if it can prove tha…

But in fact compilers do regularly prove such things as, "this function call did not touch that local variable". Escape analysis is a term related to this. I'm more of two minds about that other step, where the compiler goes like, "here in the printf call the p will be dereferenced, so it surely is non-null, so we silently optimize that other thing out where we consider the possibility of it being null". Also @joshua…

You don't need to worry about null check removal optimizations unless you do this:

    int main() {
      char *p;
      p = mmap(0, 65536, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
      // ...
      return __builtin_popcountl((uintptr_t)p);
    }
Or you do this:

    void ContinueOnError(int sig, siginfo_t *si, ucontext_t *ctx) {
      xed_decoded_inst_zero_set_mode(&xedd, XED_MACHINE_MODE_LONG_64);
      xed_instruction_length_decode(&xedd, (void *)ctx->uc_mcontext.rip, 15);
      ctx->uc_mcontext.rip += xedd.length;
    }

    int main() {
      signal(SIGSEGV, ContinueOnError);
      volatile long *x = NULL;
      printf("*NULL = %ld\n", *x);
    }

Re: The Byte Order Fiasco

#352

Earlier quoted context omitted.

char* is a allowed to alias to other pointer types.

Hm. Afsik, you are always allowed to convert _to_ a char, but _from_ is not ok in general. See i.e. [0] [0] https://gist.github.com/shafik/848ae25ee209f698763cffee272a5...

Why is it not ok to convert from a char? Some of the information in the gist is wrong. Type punning with unions for example is legal. ANSI X3.159-1988 is quite clear on that point in its aliasing rules. I've seen a lot of comments people post online saying you must use memcpy to read the bits in a float or that c++ forbids union punning but where is that written. Since if that were true every math library would break.

Re: The Byte Order Fiasco

#353
post #349

Earlier quoted context omitted.

Using the preprocessor, something like this: uint32_t swap32(uint32_t x) { ... } #if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ uint32_t swap32be(uint32_t x) { return swap32(x); } #elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ uint32_t swap32be(uint32_t x) { return x; } #else #error "Unknown endian" #endif You can make the preprocessor condition broader if you care about more compilers and more platforms. Yes, I'm makin…

Author here. The blog post has that as the naive example. The whole intention was to help people understand why we don't need to do that. Could you at least explain why you disagree if you're going to use this thread to provide the complete opposite advice?

When I read the blog post I saw this,

    #define READ32BE(p) bswap_32(*(uint32_t *)(p))
Which as you correctly state in the article, is incorrect code. We agree about this. I proposed an alternate solution, where the READ32BE would be like this:

    uint32_t read32be(const void *ptr) {
        uint32_t x;
        memcpy(&x, ptr, sizeof(x));
        return swap32be(x); // Nop on big-endian.
    }
What I like about this is that it breaks the problem down into two parts: reading unaligned data and converting byte order. The reason for this is, sometimes, you need a half of that. Some wire formats have alignment guarantees, and if you know that the alignment guarantees are compatible with your platform, you can just read the data into a buffer and then (optionally) swap the bytes in place.

Just to give an example... not too long ago I was working with legacy code that was written for MIPS. Unaligned access does not work on MIPS, so the code was already carefully written to avoid that. All I had to do was make sure that the data types were sized (e.g. replace "long" with "int32_t") and then go through and byte swap everything.

    struct Something {
        int32_t x, y;
        char name[16];
    };

    void Something_Swap(struct Something *p) {
        p->x = swap32be(p->x);
        p->y = swap32be(p->y);
    }
So it's nice to have a function like swap32be(), and "you don't have to mask and shift" I would say is true, it just depends on which compilers you want to support. I would say that a key part of being a C programmer is making a conscious decision about which compilers you want to support.

Yes, I'm aware that structs are not a great way to serialize data in general, but sometimes they're damn convenient.

Re: The Byte Order Fiasco

#354

Earlier quoted context omitted.

24 hours a day, man. I'm always byte swapping. (I'm not sure how to answer the question... what do you mean, "when?")

The entire problem of using byte swaps is that you need to use them when your native platform's byte order is different from that of the data you are reading. You know the byte order of the data. But the tricky part is, what is the byte order of the platform?

Whether it is tricky depends on what platforms you care about.

Re: The Byte Order Fiasco

#355

Earlier quoted context omitted.

The entire problem of using byte swaps is that you need to use them when your native platform's byte order is different from that of the data you are reading. You know the byte order of the data. But the tricky part is, what is the byte order of the platform?

Whether it is tricky depends on what platforms you care about.

Or, you can just follow the advice of the article, and not need to worry about it because the compiler takes care of it for you.

Re: The Byte Order Fiasco

#356

Earlier quoted context omitted.

https://news.ycombinator.com/item?id=27086928 These various ways to encode Unicode have quite a lot to do with bytes being 8-bit sized !

But Unicode itself doesn't! Anyway, it doesn't make much sense to define the size of a “byte“ as anything else then 8 bits, because that's the smallest adressable memory unit. If you need a 32 bit data type, just use one!

My very point is that we should have increased the size of the smallest addressable memory unit from 8 to 32 bits, increased again, as previous computer architectures used from 4 to 7 bits per byte. (There might be still e-mail servers around directly compatible with "non-padded" 7-bit ASCII ?)

Re: The Byte Order Fiasco

#357
post #347

Earlier quoted context omitted.

To start with, RGB (and I assume Y′CbCr ?) can be encoded in many different ways. The most common one today (still) uses 8 bits per channel , meaning that a separated 1-octet value can only define monochrome. Therefore 8-bpc RGB is a 24-bit sized format, not a 8-bit sized data one. And, by an interesting coincidence, with the arrival of "HDR", 8-bit per channel is slowly becoming obsolete (because insufficient). The…

We're debating semantics, but if I reshaped an RGB image into component arrays i.e. u8[yn][xn][3] → u8[3][yn][xn] then would you still view that as a 24-bit format? What if those 24-bit values were huffman or run-length encoded would it be an n-bit format? If your Y′CbCr luminance plane has a legal range of 16..235 and the chrominance planes are 16..240, then would it be a 23.40892 bit format?

I'm arguing about non-compressed, eventually padded data types that make learning Unicode (or any other applicable data format) easier because of the equivalence : 1 atomic unit ("character", pixel) = 1 smallest addressable unit of memory (byte). This involves byte size being at least as large as atom size.

And it's particularly important to have this property for text, because not only data is overwhelmingly stored as text (in importance, not by "weight"), but because computer programs themselves are written using text.

Re: The Byte Order Fiasco

#358

Earlier quoted context omitted.

There have been CPU architectures where the endianness at compile time isn't necessarily sufficient. I forget which, maybe it was DEC Alpha, where the CPU could flip back and forth? I can't recall if it was a "choose at boot" or a per process change.

ARM allows dynamic changing of endianess[1]. [1]: https://developer.arm.com/documentation/dui0489/h/arm-and-th...

Which nothing will be able to deal with so you might as well not bother to support it. Your compiler will also assume a fixed endianness based on the target triple.

Re: The Byte Order Fiasco

#359

Earlier quoted context omitted.

Whether it is tricky depends on what platforms you care about.

Or, you can just follow the advice of the article, and not need to worry about it because the compiler takes care of it for you.

> because the compiler takes care of it for you.

It will always be correct, but you can't just assume that the compiler will optimize the shifts into a byteswap instructions. If you look at the article you will see that it tires to no-true-scotsman that concern away by talking about a "good modern compiler".

Re: The Byte Order Fiasco

#360
I agree that in an ideal world we should just write load code using byte loads and shifts. But in the world we live in, compilers only got the ability to recognize that and emit a bswap instead in relatively recent [0] versions (compared to the age of C). And the recognition can still depend on the exact pattern used. Also, debug builds will still emit the whole shift mess, which in some cases can be annoying.

[0] https://godbolt.org/z/jMbqT86jo

Post reply on HN