171–180 of 193 posts
Earlier quoted context omitted.
Many R-L languages do the same such as German French, Hindi et al.
Wait, Hindi is neither a R-L language nor does it switch directions for text and numbers. Source: native Hindi speaker.
I was once on a contract that involved porting satellite simulation software from SPARC/Solaris to x86/RHEL. The software was in Ada95, and it didn't take much to get it to compile on the new hardware. However, ground-to-vehicle comms (which our simulator had to handle as it needed to work with the real ground control software) took place using a packed binary format (wasting as little space as possible). Data fields…
One time I had to make little- and big-endian machines talk nicely was for a message engine that ran on Intel/Linux. All the messages in the ICD were glorified packed structs that were native on the IBM/AIX side.
I used the ICD wire format as my storage format. I defined containers for all the primitive types with correct sizes and that swapped values on assignment and read (except for strings, of course). I added the appropriate pragmata to pack the underlying structs and locked down their sizes with static_asserts that would fail at compilation time until everything fit perfectly.
These classes gave me what looked like native structs on the Linux side, e.g.,
msg.foo = 3;
msg1.date = msg2.date + 1;
A cool aspect to the story was it all worked correctly the first time out of the box even though I did not have access to the AIX box during implementation. When it came time to integrate with the AIX side, all that was left to do was some low-level socket stuff to hook up to the message flow.
I’d moved to a different project full-time and worked this message engine during evenings and weekends. We got the code unpacked, so I run make check like I had been so frequently.
“Wow, you had time to write unit tests and everything?” asked one of the guys from behind my shoulder.
A moment of Zen hit.
“I didn’t have time not to.”
As long as we use the traditional network protocols and socket API it's not "dead" dead. The other name for big-endian is "network order", after all. As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing. In one instance I was reading a spec for an industry-specific protocol. At…
The same was done "back in the day". On our systems, we had a reliable transport protocol (precursor to TCP/IP) that took advantage of the fact that the network and hardware was all BE. Everyone tried to squeeze every last bit of performance out of the machines and avoiding the byte shuffling was not an insignificant performance gain. It's not surprising that people are now doing the same thing with x86-64 given its…
As long as we use the traditional network protocols and socket API it's not "dead" dead. The other name for big-endian is "network order", after all. As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing. In one instance I was reading a spec for an industry-specific protocol. At…
> they've designed the spec to be whatever GCC on x86-64 Linux machine would do to layout C the structures in memory. In a previous job, the software I worked on had a "file format" that just consisted of dumping a raw array of structs to disk. It was obviously pretty fast, but it made cringe a lot. What's worse, the compiler had flags to control if and how much struct padding to use, so reading a data file with a bi…
[1] Unless you happen to access a field spanning two cache lines and miss, in which case the padded version would require accessing that second cacheilne anyway.
Big endian is dead on the client. WebGL exposes endianness and happened at a time when almost all systems running browsers are little-endian. This makes the big-endian provisions of the spec a dead letter. Web devs don't need to test on big-endian systems and would have a hard time finding big-endian systems to test with even if they wanted to. It's safe to assume, therefore, that there's a mass of Web content that o…
You can not only assume it, but I can also say with certainty it exists. Emscripten hastened this by letting site owners compile little-endian code into little-endian asm.js. For example, the WhatsApp Web QR code generator won't run on a BE platform; it's an Emscriptenized blob of code backing it that assumes the typed array it stores into has LE orientation. TenFourFox and Leopard Webkit both emulate little-endian t…
I'm pretty sure many other fundamental protocols of the Web don't work so well on a non-2's-complement, non-8-bit-byte machine either.
I find the the terms big endian and little endian confusing, feels like they should be switched around. So the one that ends big (most significant byte last) should be big endian and the one that ends little should be little endian. But it's actually the opposite. It makes it difficult to remember which is which.
Little endian puts the
Least significant part at the
Lowest address(offset)Earlier quoted context omitted.
> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen. The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you defi…
I have to disagree with your “No no no”. It all depends on requirements. When you’re on a PC, and working with files that you can reasonably expect will be exchanged (like doc, xls or pdf) — then your “No no no” heuristic is absolutely correct. As you pointed out, differences between compilers and between e.g. x86/amd64 are very likely to render these formats incompatible. But when you’re working with files that only…
If:
* you're working with files that only your software will access AND
* you control, and understand, the CPU architecture of any machine that will touch that data structure and you KNOW that it will never change for the entire lifetime of that piece of data AND
* you always use the same version, or an ABI-compatible future version, of the same compiler that you KNOW will always lay out data the same way and you know what that way is AND
* you either don't need to touch this data structure in other programming languages, or you KNOW that this won't be an issue (for example because your programming language implementation was written in C, compiled against the same version of the compiler, and has an FFI that understands C structs)
THEN, you may proceed to mmap structs into memory. In practice these constraints are fairly commonplace; for example, NetBSD wscons device drivers report HID events which are specified in a struct, and because the HID events are likely never to leave the originating machine, only be passed from kernel space to user space, it makes sense to simply read(2) them straight into the struct.
And there's certainly nothing wrong with snarfing a large file into a char[], but when it comes time to extract meaning from it, unless you are absolutely sure that the underlying assumptions regarding data layout will never ever change, it will only cause marginal harm to do everything in byte offsets and shift and OR bytes to yield final values -- and this is the best portable way to access the data therein, agnostic of details about the compiler and CPU architecture.
In general it's a good idea to err on the side of caution by default, profile, and then optimize the hot paths as necessary bearing the constraints you're assuming in mind (and perhaps documenting them for good measure).
Maybe my initial statement was too strong, but I shudder whenever I see comments of the form "It's so convenient to just mmap() that sucker into a struct and access the fields!" Because they've never had to deal with the consequences of trying to access a struct from a Microsoft compiler serialized to disk, and finding out that GNU compilers have a quite different notion of how structure members are to be laid out in memory...
Little endian is also better for multi-word addition/subtraction. Your first operand will be the low word, which your pointer conveniently already indexes. This was a classic trick in the days of 8 and 16 bit systems, and assembly code for modern modular crypto systems still uses it. Really I think consensus is that the industry focus on big endian systems was actually a mistake. It prioritized programmer comfort in…
Earlier quoted context omitted.
Using the x86-64 C struct layout as the serialization format is increasingly common in financial trading protocols. Skipping all the bit twiddling has a huge performance impact at high message rates. I once compared two feeds covering the same basic data, where the major difference was one had a complicated serialization and one was C struct layout. The former needed two 16 core servers just to process I/O and deseri…
Regarding bring lazy on the serialization... yeah packed structures make sense. But excess padding is still wasteful, and endianness is only a bswap away, which is 1-2 cycles apiece and so won't exactly break the latency budget (per Agner Fog's wonderful pdf, looking at Haswell).
Earlier quoted context omitted.
You can not only assume it, but I can also say with certainty it exists. Emscripten hastened this by letting site owners compile little-endian code into little-endian asm.js. For example, the WhatsApp Web QR code generator won't run on a BE platform; it's an Emscriptenized blob of code backing it that assumes the typed array it stores into has LE orientation. TenFourFox and Leopard Webkit both emulate little-endian t…
IMHO, this is a broken promise about how the Web was supposed to be platform-independent. I get the feeling the general attitude on this thread is that such platforms are unimportant, but that doesn't mean it's not a broken promise. I'm pretty sure many other fundamental protocols of the Web don't work so well on a non-2's-complement, non-8-bit-byte machine either.
WebGL's endian specification was the beginning of the end, after which folks started saying certain implementational details weren't worth caring about, and this just finishes it off. And that's not what was being sold in the beginning.