Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is big-endian dead?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 193 posts

Re: Ask HN: Is big-endian dead?

#121

Post by Const-me »

bitwize wrote: post #61
tlb wrote: post #4

Earlier quoted context omitted.

Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. You can even deliver those files to web browsers and use JS's TypedArray to get random access into them. (That requires a bit more than simply little-endian. It requires struct alignment and floating point formats to be the same. But with only a…

> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen. The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you defi…

I have to disagree with your “No no no”. It all depends on requirements.

When you’re on a PC, and working with files that you can reasonably expect will be exchanged (like doc, xls or pdf) — then your “No no no” heuristic is absolutely correct. As you pointed out, differences between compilers and between e.g. x86/amd64 are very likely to render these formats incompatible.

But when you’re working with files that only your software will access, unpacking it byte by byte will slow down IO by a huge factor compared to both mmap, and read/write of large blocks (the latter will likely translate to DMA i.e. the CPU will be free to do something else). When that’s the case (like for most videogames, even for PC ones), you don’t want to pay that performance penalty, you want to get that data in the memory ASAP.

Re: Ask HN: Is big-endian dead?

#122

Post by gumby »

samat wrote: post #37
gumby wrote: post #32

> Little-endian is slightly more confusing for humans... I think you meant to write “Little-endian is slightly more confusing for humans who use left-to-right languages ” as all the R-L languages also put the least significant digit on the right.

Not true. In arabic languages you write text R-L, but numbers are L-R.

Exactly: you encounter the least significant digit first as you read.

Re: Ask HN: Is big-endian dead?

#123

Post by kentonv »

bitwize wrote: post #113
kentonv wrote: post #94

Earlier quoted context omitted.

That's incorrect. When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing". Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able ac…

> When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing". Assuming little-endian CPU arch. It's followed by a byte reorder on big-endian architectures. (And you assume all Windows instances are little-endian, which probably-is-but-may-not-be the case.) You made the dec…

> Assuming little-endian CPU arch. It's followed by a byte reorder on big-endian architectures.

Basically all common CPUs are LE.

(And basically all BE CPUs have dedicated instructions for reading LE data. It's true I haven't yet added the inline assembly to use those instructions in Cap'n Proto's reference implementation, but that's only because no one actually cares about these architectures.)

So in basically all real use, there's no machine-instruction-level difference between accessing a field of a capnp struct and accessing a field of a C struct. If the instructions are identical, then how can you say one is "deferred parsing" and the other isn't? What meaning does any such distinction have?

> There's a performance cost, but hopefully you're only doing serialization/deserialization when you intend to hit the disk or wire to read/write into/out of your struct.

Disk is usually cached, meaning it's already in physical memory and you're wasting time making a copy rather than using the data in-place.

Over the network, within a datacenter, bandwidth is basically infinite (in that your CPU probably can't process bytes as fast as your network interface can). Time spent serializing and parsing is very real and wasteful. I've seen servers spending 30% or more of their CPU time parsing protobufs.

Over the long-haul internet, perhaps the CPU time spent parsing/serializing is not as relevant compared to the time spent transmitting. Still, I'd rather spend my CPU cycles elsewhere -- like in a dedicated compression algorithm -- rather than twiddling bytes needlessly in a parser.

> handling arbitrary binary formats

Yes, we all agree that some binary formats can't be handled any other way. But if you're in control of the format you use, then you can design it in a way that doesn't require a "bag of bytes" model, and your code can be much simpler and more adaptable as a result.

Re: Ask HN: Is big-endian dead?

#124

Post by gumby »

alaaibrahim wrote: post #49
amaccuish wrote: post #44

Earlier quoted context omitted.

No. Arabic numbers are always L-R...

Not exctly accurate, for the part of the number that is Also for the part that is L-R, that is not the rule, as some people still read all the number as R-L (actually in a lot of historical documents that was the case), so the would read 1925 as five and twenty and nine hundred and a thousand. Where is now most people would read it as a thousand, and nine hundred, and five and twenty.

Many R-L languages do the same such as German French, Hindi et al.

Re: Ask HN: Is big-endian dead?

#125

Post by gumby »

IshKebab wrote: post #80
gumby wrote: post #32

> Little-endian is slightly more confusing for humans... I think you meant to write “Little-endian is slightly more confusing for humans who use left-to-right languages ” as all the R-L languages also put the least significant digit on the right.

Honestly I find little endian less confusing. The only confusing thing is how little endian value are written in hex because within each byte they are written as big endian. Other than that little endian makes more sense. The fact that Arabic numbers are written in big endian is weird. Nobody questions it because it's all anyone is taught.

They aren’t Arabic BTW, they are Hindu positional digits (look at the characterforms you’ll see the connection) which arrived in Europe via the Arabs, hence the name — in Arabic, different character forms are used.

Re: Ask HN: Is big-endian dead?

#126

Post by kentonv »

kuschku wrote: post #112
kentonv wrote: post #108

Earlier quoted context omitted.

Yes, Cap'n Proto is careful to require that the data is aligned. (Protobuf, on the other hand, fundamentally doesn't allow for multi-byte loads in the first place since integers use variable-width encoding, so alignment is irrelevant there.)

Yes. For raw cap'n'proto messages. But when writong custom file formats you might end up having cap'n'proto data embedded at an unaligned offset.

Sure, when layering on top of a non-zero-copy serialization, you may be forced to do a memcpy() of your data upfront to get alignment.

Re: Ask HN: Is big-endian dead?

#127

Post by gumby »

astrodust wrote: post #57
gumby wrote: post #31

Earlier quoted context omitted.

FWIW every one of the machines you cited (“Sparc, old MIPS, old PPC, DEC Alpha, etc”) post dated the formation of the Internet (the arpanet transitioning to TCP); the Internet protocols just followed existing arpanet practice. Which was due to big-endian processors being common, but the dominant networked machine of that era was the PDP-10.

The PDP successor, the VAX, was strangely little-endian.

PDP-11 had a little endian architecture while big endian machines like the PDP-6/PDP-10 predominated on the net.

To make life more interesting, the later PDP-10/20 mainframes used PDP-11 minicomputers as front end processors and often as network processors so byte-swapping was the norm. Luckily the PDP-10 allowed bytes ranging from 1-36 bits wide - “byte” had not yet standardized on 8 bits

Re: Ask HN: Is big-endian dead?

#128

Post by zerokernel »

kentonv wrote: post #123
bitwize wrote: post #113

Earlier quoted context omitted.

> When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing". Assuming little-endian CPU arch. It's followed by a byte reorder on big-endian architectures. (And you assume all Windows instances are little-endian, which probably-is-but-may-not-be the case.) You made the dec…

> Assuming little-endian CPU arch. It's followed by a byte reorder on big-endian architectures. Basically all common CPUs are LE. (And basically all BE CPUs have dedicated instructions for reading LE data. It's true I haven't yet added the inline assembly to use those instructions in Cap'n Proto's reference implementation, but that's only because no one actually cares about these architectures.) So in basically all r…

> but that's only because no one actually cares about these architectures.

This. Optimizing performance on BE is a waste of everyone's time and resources.

Re: Ask HN: Is big-endian dead?

#129

Post by takeda »

I read and so far all arguments for little endian I see are because the CPUs adopted it so it is more efficient to keep data the same way.

What are the other benefits of little endian, because in terms of readability big endian makes most sense.

I could understand that having things in reverse could be somehow more efficient, but why do it reverse in groups of 8 bits instead having all bits reversed?

Re: Ask HN: Is big-endian dead?

#130

Post by zerokernel »

kentonv wrote: post #94
bitwize wrote: post #89

Earlier quoted context omitted.

Both of those appear to work by deferring the parsing step to access time. You're still treating the thing as a bag of bytes and unpacking stuff out of it bytewise.

That's incorrect. When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing". Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able ac…

> Byte-by-byte parsing is a valid way to do parsing but not the only way. Byte-by-byte parsers tend to be slow and -- arguably, more importantly -- overly complex and rigid. It is, for example, usually very hard to do "random access" with a byte-by-byte parser, because allowing out-of-order parsing tends to blow the code complexity through the roof.

I have to agree here by experiences past. If the format in question has a chance of being performance sensitive, don't use FSM-based encodings [1]. It is inordinately difficult to optimize parsing these encodings even if you only have to handle tiny subsets, and it still won't be fast. A format like msgpack which prides itself on being very fast may be fast compared to JSON and other ways to express essentially arbitrary structures, but is DEAD SLOW compared to any direct encoding (be it a dedicated encoding you developed in literally a few hours or something like capnproto).

[1] Obviously, considering an encoding more complex than FSM means that you're an idiot and your application will almost certainly have security vulnerabilities related to the format in the future.

Post reply on HN