Ask HN: Is big-endian dead?

> Whether you call the value 1 "bit 7", "bit 8", "bit 1" or "bit 0" is just, pardon the pun, word semantics.

It's extremely important to your mental model to understand how the bits are arranged, or left shift (1 0 0 0 0 0 0 0

Which would left shift to:

0 0 0 0 0 0 0 0

Which would not be equal to 2.

That's incorrect. When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing".

Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able accessor functions to manipulate these structures which do pointer arithmetic similar to what a compiler would generate for accessing a struct. The end result is that accessing primitive fields from a Cap'n Proto struct is essentially identical in terms of machine instructions to accessing fields of a C struct.

To call that "deferring the parsing to access time" does not make sense.

(Note that for pointers, there are more differences: namely, because data will not always be loaded at the same address, pointers need to be relative rather than absolute. They also need to be bounds-checked for security. This adds a few instructions to pointer accesses, but those instructions look nothing like traditional "parsing" and certainly aren't byte-by-byte operations.)

Your statement earlier:

> The correct way to handle binary data is to unpack it into the struct byte by byte.

This is inaccurate. Byte-by-byte parsing is a valid way to do parsing but not the only way. Byte-by-byte parsers tend to be slow and -- arguably, more importantly -- overly complex and rigid. It is, for example, usually very hard to do "random access" with a byte-by-byte parser, because allowing out-of-order parsing tends to blow the code complexity through the roof.

On the other hand, with Cap'n Proto and similar approaches, you can trivially mmap() a very large data structure and traverse it randomly, and it "just works".

(Disclosure: I'm the author of Cap'n Proto, as well as the author of the first open source release of Google's Protocol Buffers, which does byte-by-byte binary parsing.)

Everything is consistent in the big (down to the nybble and bit) endian view.

The number 0x12345678 is actually the nybbles 1 2 3 4 ... which are the bits 0001 0010 0011 0100 and so on.

In the hex dump 12 34 56 78 we just understand the bytes to be big-endian also at the nybble level (and bit also).

That is to say the "1" can be understood to be at the lower "nybble address", relative to the "2".

If that buffer is sent over a serial communication channel or network, the bits actually go in that order 0001 0010 0011. The 1 nybble goes out first, as 0001 (three zeros out the door, then a one), then the 2 nybble and so on.

Big endian is dead on the client. WebGL exposes endianness and happened at a time when almost all systems running browsers are little-endian. This makes the big-endian provisions of the spec a dead letter. Web devs don't need to test on big-endian systems and would have a hard time finding big-endian systems to test with even if they wanted to. It's safe to assume, therefore, that there's a mass of Web content that only works if the browser exposes little-endian behavior to JS. It's not performance-wise competitive for a WebGL system to present little-endian behavior on big-endian hardware.

Therefore, it won't be feasible to try to re-introduce big endian to systems that need to be competitive at rendering the Web.

Big endian will stay alive for the time being on home routers and Sparc servers, and probably for a long time on IBM z systems.

The cost–benefit of software accommodating big-endian systems will look increasingly bad with legacy enterprise servers imposing a negative externality on everyone else. (E.g. refusal to consider bitcasts between SIMD vectors of the same bit width but different number of lanes as portable/safe operation in language design.)

They're replacing MIPS with ARM as fast as they can but I wonder which endian they're running. Certainly ARM servers will run LE but I wonder if they will bother to produce a BE SDK for networking to ease porting from MIPS BE.