We had this same thread 997 days ago: https://news.ycombinator.com/item?id=9451284
If big endian was effectively dead back then, I guess it's really dead now.
Ask HN: Is big-endian dead?
101–110 of 193 posts
Re: Ask HN: Is big-endian dead?
#102As long as we use the traditional network protocols and socket API it's not "dead" dead. The other name for big-endian is "network order", after all. As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing. In one instance I was reading a spec for an industry-specific protocol. At…
> The other name for big-endian is "network order", after all.
I wonder where this started... I always suspected it was Sun with their "The Network is the Computer" motto, and of course they would define "network byte order" as what they used.
Today, it drives me crazy that we're constantly swapping before sending across the network and then swapping again when it's received.
Re: Ask HN: Is big-endian dead?
#103Earlier quoted context omitted.
Maybe that mattered on serial connections, but on a 10Gbit switch you're talking nonsense.
Only if all you care about is that 10Gbit is a lot of throughput, and latencies don't bother your application. If a 10Gbit network is being used precisely because 1Gbit didn't have sufficiently low latency, then it matters.
But more than that, the encoding used in 10GBASE-T doesn't allow decoding individual bytes. For the usual copper standard, it's 4 pairs with 7 bits per symbol, and the skew between lanes means you can't count on corresponding symbols arriving at the same time. And then there's a reed-solomon block code to correct errors, which has to be decoded all at once. The optical standards use 64b/66b coding, which means you receive blocks of 66 bits and decode 64 out of them (which prevents having too many zeros in a row which can disrupt timing.)
Re: Ask HN: Is big-endian dead?
#104As long as we use the traditional network protocols and socket API it's not "dead" dead. The other name for big-endian is "network order", after all. As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing. In one instance I was reading a spec for an industry-specific protocol. At…
Using the x86-64 C struct layout as the serialization format is increasingly common in financial trading protocols. Skipping all the bit twiddling has a huge performance impact at high message rates. I once compared two feeds covering the same basic data, where the major difference was one had a complicated serialization and one was C struct layout. The former needed two 16 core servers just to process I/O and deseri…
"The former needed two 16 core servers just to process I/O and deserialize the messages; the latter used 1 core on a single server."
I'm genuinely curious; what was the 'former' protocol? Was it encoded using FAST or zlib or something?
Re: Ask HN: Is big-endian dead?
#105Earlier quoted context omitted.
Both of those appear to work by deferring the parsing step to access time. You're still treating the thing as a bag of bytes and unpacking stuff out of it bytewise.
That's incorrect. When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing". Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able ac…
> The end result is that accessing primitive fields from a Cap'n Proto struct is essentially identical in terms of machine instructions to accessing fields of a C struct.
As long as the Cap'n Proto or ProtoBuf data is properly aligned within of your custom file format, or you might end up with unusual slowness: https://blogs.msdn.microsoft.com/oldnewthing/20150116-00/?p=...
Re: Ask HN: Is big-endian dead?
#106As long as we use the traditional network protocols and socket API it's not "dead" dead. The other name for big-endian is "network order", after all. As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing. In one instance I was reading a spec for an industry-specific protocol. At…
Using the x86-64 C struct layout as the serialization format is increasingly common in financial trading protocols. Skipping all the bit twiddling has a huge performance impact at high message rates. I once compared two feeds covering the same basic data, where the major difference was one had a complicated serialization and one was C struct layout. The former needed two 16 core servers just to process I/O and deseri…
Which also happens to be why we used binary document formats first then moved to XML too.
Re: Ask HN: Is big-endian dead?
#107If you work with some industrial machines via serial protocols it is very much alive.
Re: Ask HN: Is big-endian dead?
#108Earlier quoted context omitted.
That's incorrect. When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing". Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able ac…
> The end result is that accessing primitive fields from a Cap'n Proto struct is essentially identical in terms of machine instructions to accessing fields of a C struct. As long as the Cap'n Proto or ProtoBuf data is properly aligned within of your custom file format, or you might end up with unusual slowness: https://blogs.msdn.microsoft.com/oldnewthing/20150116-00/?p=...
Yes, Cap'n Proto is careful to require that the data is aligned.
(Protobuf, on the other hand, fundamentally doesn't allow for multi-byte loads in the first place since integers use variable-width encoding, so alignment is irrelevant there.)
Re: Ask HN: Is big-endian dead?
#109Earlier quoted context omitted.
Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. You can even deliver those files to web browsers and use JS's TypedArray to get random access into them. (That requires a bit more than simply little-endian. It requires struct alignment and floating point formats to be the same. But with only a…
> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen. The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you defi…
[deleted]
Re: Ask HN: Is big-endian dead?
#110little endian is only confusing for humans because we very inconsistently use RTL vs LTR semantics.