Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is big-endian dead?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 193 posts

Re: Ask HN: Is big-endian dead?

#111

Post by twic »

usefulcat wrote: post #104
scott00 wrote: post #70

Earlier quoted context omitted.

Using the x86-64 C struct layout as the serialization format is increasingly common in financial trading protocols. Skipping all the bit twiddling has a huge performance impact at high message rates. I once compared two feeds covering the same basic data, where the major difference was one had a complicated serialization and one was C struct layout. The former needed two 16 core servers just to process I/O and deseri…

"The former needed two 16 core servers just to process I/O and deserialize the messages; the latter used 1 core on a single server." I'm genuinely curious; what was the 'former' protocol? Was it encoded using FAST or zlib or something?

I would guess FIX/FAST. For a concrete example, have a read of the specifications for Eurex's market data protocols:

http://www.eurexchange.com/exchange-en/technology/t7/system-...

Specifically, compare the Enhanced Market Data Interface protocol (described in "T7 Market and Reference Data Interfaces") and the Enhanced Order Book Interface protocol. EMDI is FAST, a morass of tags, stop bits, presence bitmaps, and who knows what else (i don't). EOBI is structs.

Re: Ask HN: Is big-endian dead?

#112

Post by kuschku »

kentonv wrote: post #108
kuschku wrote: post #105

Earlier quoted context omitted.

> The end result is that accessing primitive fields from a Cap'n Proto struct is essentially identical in terms of machine instructions to accessing fields of a C struct. As long as the Cap'n Proto or ProtoBuf data is properly aligned within of your custom file format, or you might end up with unusual slowness: https://blogs.msdn.microsoft.com/oldnewthing/20150116-00/?p=...

Yes, Cap'n Proto is careful to require that the data is aligned. (Protobuf, on the other hand, fundamentally doesn't allow for multi-byte loads in the first place since integers use variable-width encoding, so alignment is irrelevant there.)

Yes. For raw cap'n'proto messages. But when writong custom file formats you might end up having cap'n'proto data embedded at an unaligned offset.

Re: Ask HN: Is big-endian dead?

#113

Post by bitwize »

kentonv wrote: post #94
bitwize wrote: post #89

Earlier quoted context omitted.

Both of those appear to work by deferring the parsing step to access time. You're still treating the thing as a bag of bytes and unpacking stuff out of it bytewise.

That's incorrect. When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing". Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able ac…

> When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing".

Assuming little-endian CPU arch. It's followed by a byte reorder on big-endian architectures. (And you assume all Windows instances are little-endian, which probably-is-but-may-not-be the case.) You made the decision to optimize for what you consider the common case, but it does not generalize without added translation code to all cases. You may have hidden the translation code behind CapnProto's generated accessors, but CapnProto's structs are translation-free the way AWS Lambda is "serverless".

> To call that "deferring the parsing to access time" does not make sense.

Except you are deferring translation work (like byte reordering) to access time. Either that or you're hiding it in the serialization APIs. Again, it's like "serverless" computing: just because you've hidden it doesn't mean it's gone away.

> Byte-by-byte parsing is a valid way to do parsing but not the only way. Byte-by-byte parsers tend to be slow and -- arguably, more importantly -- overly complex and rigid.

There's a performance cost, but hopefully you're only doing serialization/deserialization when you intend to hit the disk or wire to read/write into/out of your struct. All in-memory processing happens in whatever endianness and alignment makes your CPU and compiler happy.

There's nothing "overly complex and rigid" about understanding a binary format as a bag of bytes and fetching scalar values (including offsets into the data structure) from it accordingly. This is how shit gets done when it comes to portably handling arbitrary binary formats. CapnProto can score a few wins by assuming things about the target CPU/compiler, restricting the binary format to conform to some of those assumptions, and papering over the rest with code hidden behind some of its APIs. But it's not a general solution to the problem of extracting meaning from an arbitrary hunk of bytes that may or may not have come from a CapnProto-conformant application.

Re: Ask HN: Is big-endian dead?

#114

Post by twic »

civility wrote: post #102
rdtsc wrote: post #51

As long as we use the traditional network protocols and socket API it's not "dead" dead. The other name for big-endian is "network order", after all. As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing. In one instance I was reading a spec for an industry-specific protocol. At…

> The other name for big-endian is "network order", after all. I wonder where this started... I always suspected it was Sun with their "The Network is the Computer" motto, and of course they would define "network byte order" as what they used. Today, it drives me crazy that we're constantly swapping before sending across the network and then swapping again when it's received.

This may be the best ensemble of answers you'll get to that question:

https://retrocomputing.stackexchange.com/questions/2652/when...

The internet was big-endian from the start, and that was probably because DEC machines were big-endian.

Re: Ask HN: Is big-endian dead?

#115

Post by bitwize »

kuschku wrote: post #112
kentonv wrote: post #108

Earlier quoted context omitted.

Yes, Cap'n Proto is careful to require that the data is aligned. (Protobuf, on the other hand, fundamentally doesn't allow for multi-byte loads in the first place since integers use variable-width encoding, so alignment is irrelevant there.)

Yes. For raw cap'n'proto messages. But when writong custom file formats you might end up having cap'n'proto data embedded at an unaligned offset.

Now that Cap'n Proto exists, why would you want to handle binary data any other way? Simply standardize on Cap'n Proto across your entire application, and problem solved :)

Re: Ask HN: Is big-endian dead?

#116

Post by zlynx »

macintux wrote: post #60
drewg123 wrote: post #12

Earlier quoted context omitted.

Small pedantic correction from a former Alpha kernel hacker: DEC Alpha was Big Endian only for some rare Cray systems. Everything else was Little Endian.

Many, many years ago I worked for Progeny, and we contracted with HP to help bring Debian to Itanium. We quickly discovered that a fair bit of Linux software (or at least Debian packaging of same) only worked properly on 64-bit DEC Alpha because the architecture was Little Endian; 32-bit pointers would typically "just work(™)" because the code was grabbing the first 32 bits, and the other half of the pointer addresse…

HP wanted to replace PA-RISC and that was big-endian. Intel of course wanted little-endian. That's why Itanium was selectable.

Re: Ask HN: Is big-endian dead?

#117

Post by microtherion »

api wrote: post #3
tlb wrote: post #2

I used to use IETF's network order (big-endian) for everything on disk or over the wire, but I switched to little-endian around 2008. If I ever have to port to a big-endian machine I'll deal with it then. I haven't regretted it yet.

Big-endian is the canonical form for storage and the wire because back when the Internet was designed most "pro" machines were big-endian: Sparc, old MIPS, old PPC, DEC Alpha, etc. All these are dead or dying now. It's easy enough to deal with BE files and protocols by just swapping bytes. I'm referring to hardware architectures. Are there still any big-endian chips out there? Does it still make sense to support big-…

> back when the Internet was designed most "pro" machines were big-endian: Sparc, old MIPS, old PPC, DEC Alpha, etc

You kids sure know how to make someone feel really old... Much of the internet design predates the 68000, which in turn predates the RISC architectures you cite by half a decade and more.

Re: Ask HN: Is big-endian dead?

#118

Post by kuschku »

bitwize wrote: post #115
kuschku wrote: post #112

Earlier quoted context omitted.

Yes. For raw cap'n'proto messages. But when writong custom file formats you might end up having cap'n'proto data embedded at an unaligned offset.

Now that Cap'n Proto exists, why would you want to handle binary data any other way? Simply standardize on Cap'n Proto across your entire application, and problem solved :)

Baaad idea. Quassel standardized years ago on Qt’s data serialization everywhere, and it’s become a major issue now.

Re: Ask HN: Is big-endian dead?

#119

Post by pipio21 »

bitwize wrote: post #61
tlb wrote: post #4

Earlier quoted context omitted.

Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. You can even deliver those files to web browsers and use JS's TypedArray to get random access into them. (That requires a bit more than simply little-endian. It requires struct alignment and floating point formats to be the same. But with only a…

> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen. The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you defi…

This makes no sense in lots of applications. Today computers have hardware acceleration for moving chunks of data. Big chucks of data, unpack things byte by byte makes not sense on things that have to be efficient. It is like 20.000 times more efficient to do things with blocks.

If you move text around you probably don't care about efficiency and can do it. If you care about efficiency and price(using cheap components) it is a very bad solution.

We use in all our serial-deserial a very fine grained controlled C library and it has been working flawlessly for years. This library is interfaced with C++, java, objC, swift,clojure, rust...

Re: Ask HN: Is big-endian dead?

#120

Post by scott00 »

usefulcat wrote: post #104
scott00 wrote: post #70

Earlier quoted context omitted.

Using the x86-64 C struct layout as the serialization format is increasingly common in financial trading protocols. Skipping all the bit twiddling has a huge performance impact at high message rates. I once compared two feeds covering the same basic data, where the major difference was one had a complicated serialization and one was C struct layout. The former needed two 16 core servers just to process I/O and deseri…

"The former needed two 16 core servers just to process I/O and deserialize the messages; the latter used 1 core on a single server." I'm genuinely curious; what was the 'former' protocol? Was it encoded using FAST or zlib or something?

I was consuming the feed in question via an API provided by the vendor and didn't have direct knowledge of the wire protocol, but based on how it was described to me during the sales process it was probably either FAST or a proprietary protocol with a similar design.
Post reply on HN