Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is big-endian dead?

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 193 posts

Re: Ask HN: Is big-endian dead?

#141

Post by phkahler »

Const-me wrote: post #121
bitwize wrote: post #61

Earlier quoted context omitted.

> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen. The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you defi…

I have to disagree with your “No no no”. It all depends on requirements. When you’re on a PC, and working with files that you can reasonably expect will be exchanged (like doc, xls or pdf) — then your “No no no” heuristic is absolutely correct. As you pointed out, differences between compilers and between e.g. x86/amd64 are very likely to render these formats incompatible. But when you’re working with files that only…

>> It all depends on requirements.

Requirements change.

Re: Ask HN: Is big-endian dead?

#142

Post by ycombobreaker »

scott00 wrote: post #70
rdtsc wrote: post #51

As long as we use the traditional network protocols and socket API it's not "dead" dead. The other name for big-endian is "network order", after all. As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing. In one instance I was reading a spec for an industry-specific protocol. At…

Using the x86-64 C struct layout as the serialization format is increasingly common in financial trading protocols. Skipping all the bit twiddling has a huge performance impact at high message rates. I once compared two feeds covering the same basic data, where the major difference was one had a complicated serialization and one was C struct layout. The former needed two 16 core servers just to process I/O and deseri…

Regarding bring lazy on the serialization... yeah packed structures make sense. But excess padding is still wasteful, and endianness is only a bswap away, which is 1-2 cycles apiece and so won't exactly break the latency budget (per Agner Fog's wonderful pdf, looking at Haswell).

Re: Ask HN: Is big-endian dead?

#143

Post by phkahler »

tinus_hn wrote: post #20

Let’s hope so because one of these is not better than the other but one is much better than two.

I was annoyed reading the RISC-V docs and seeing that they allow implementations to go either way. Sometimes it's better to just make a choice and tell people what it is.

Re: Ask HN: Is big-endian dead?

#144

Post by ycombobreaker »

astrodust wrote: post #58
kazinator wrote: post #48

Earlier quoted context omitted.

Big-endian means that am ultra-high-speed routing fabric can receive the most significant part of an address address and possibly start making a routing decision before the other bytes of the frame have arrived. Hands down no brainer. Even the fact that headers come before payloads is a kind of "big endian", as is the fact that important information tends to occur earlier in headers. Look at a basic Ethernet frame. T…

Maybe that mattered on serial connections, but on a 10Gbit switch you're talking nonsense.

It absolutely matters for low-latency applications. Maybe not your use case, but it's not nonsense.

Re: Ask HN: Is big-endian dead?

#145

Post by Qwertious »

phkahler wrote: post #141
Const-me wrote: post #121

Earlier quoted context omitted.

I have to disagree with your “No no no”. It all depends on requirements. When you’re on a PC, and working with files that you can reasonably expect will be exchanged (like doc, xls or pdf) — then your “No no no” heuristic is absolutely correct. As you pointed out, differences between compilers and between e.g. x86/amd64 are very likely to render these formats incompatible. But when you’re working with files that only…

>> It all depends on requirements. Requirements change.

Requirements can change, sometimes unexpectedly. But if you "know" for a fact that they won't, then doing the hypothetically more-portable thing at the expense of today's performance, is just plain overengineering.

Re: Ask HN: Is big-endian dead?

#147

Post by jsjohnst »

Qwertious wrote: post #145
phkahler wrote: post #141

Earlier quoted context omitted.

>> It all depends on requirements. Requirements change.

Requirements can change, sometimes unexpectedly. But if you "know" for a fact that they won't, then doing the hypothetically more-portable thing at the expense of today's performance, is just plain overengineering.

You never know something like that “as a fact”.

Re: Ask HN: Is big-endian dead?

#148

Post by PeCaN »

TheOtherDave wrote: post #138
userbinator wrote: post #137

Earlier quoted context omitted.

Yes, LE is actually the more logical and elegant one, since bit n has weight 2^n, and byte n has a weight 256^n. The BE equivalents introduce an extra length- term into the equation (length-n). I've always found LE vs BE to be somewhat like 0- vs 1-based array indices; one is more "conventional" to a human, but the other doesn't require a lot of contortions when expressed algebraically or algorithmically. (With 0-bas…

Or you could just decrement base.

This also conveniently allows you to store the length at the "real" base.

    0 array length

Re: Ask HN: Is big-endian dead?

#149

Post by garmaine »

kazinator wrote: post #47
ajross wrote: post #8

Little endian is also better for multi-word addition/subtraction. Your first operand will be the low word, which your pointer conveniently already indexes. This was a classic trick in the days of 8 and 16 bit systems, and assembly code for modern modular crypto systems still uses it. Really I think consensus is that the industry focus on big endian systems was actually a mistake. It prioritized programmer comfort in…

OTOH, big-endian is better for comparison; you can tell when one integer is larger than another by looking at as little as just one bit, from the most significant end.

To follow up on this, I recently implemented a packed vector format. As a LE advocate I surprised myself by ordering the packing in BE order — because that let me simply byte-compare vector encodings to compare the underlying vectors, which was a huge performance increase.

Data structures which are frequently compared, e.g. to be sorted, benefit from BE ordering. Just binary compare them as byte strings, usually an efficient inline operation.

If I were implementing a packed wire format for SQL data, for example, I’d probably do it in BE order.

Post reply on HN