Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is big-endian dead?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 193 posts

Re: Ask HN: Is big-endian dead?

#61

Post by bitwize »

tlb wrote: post #4
api wrote: post #3

Earlier quoted context omitted.

Big-endian is the canonical form for storage and the wire because back when the Internet was designed most "pro" machines were big-endian: Sparc, old MIPS, old PPC, DEC Alpha, etc. All these are dead or dying now. It's easy enough to deal with BE files and protocols by just swapping bytes. I'm referring to hardware architectures. Are there still any big-endian chips out there? Does it still make sense to support big-…

Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. You can even deliver those files to web browsers and use JS's TypedArray to get random access into them. (That requires a bit more than simply little-endian. It requires struct alignment and floating point formats to be the same. But with only a…

> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file.

No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen.

The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you define a struct in C(++), there are not just endianness issues, but implementation-dependent issues of padding and alignment you have to consider. Recently, most compilers on most architectures standardized on self-alignment rules for all primitive types except char, but this is not guaranteed by the standard, it will bite you in the ass when you least expect it, and it will be decades yet before all the C code in the world is displaced by a single-implementation language like Rust.

The best way to write portable code that will work as intended is to treat all data coming from disk or over the wire as a bag of bytes, and not attempt to alias it to a struct.

Re: Ask HN: Is big-endian dead?

#62

Post by kazinator »

jfries wrote: post #28
loeg wrote: post #22

Earlier quoted context omitted.

So the dump is ordered backwards, from top of memory to bottom? That seems harder for humans than little endian integers.

I don't understand what you mean. Line breaks can be inserted where ever suitable. Whether the bytes are listed left-to-right or right-to-left makes no difference.

A dump of memory (several kilobytes, megabytes or whatever) is inherently big endian: it proceeds from the base address and goes up.

It is counterintuitive to swap pieces of it into some locally opposite order.

Yet, that's what has to be done so that numbers are readable.

That's why "od" has modes for that.

  $ od -tx1  /bin/ls | head -1
  0000000 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  $ od -tx2  /bin/ls | head -1
  0000000 457f 464c 0102 0001 0000 0000 0000 0000
  $ od --endian=big -tx2  /bin/ls | head -1  # GNU extension, probably.
  0000000 7f45 4c46 0201 0100 0000 0000 0000 0000

Re: Ask HN: Is big-endian dead?

#63

Post by dogecoinbase »

astrodust wrote: post #58
kazinator wrote: post #48

Earlier quoted context omitted.

Big-endian means that am ultra-high-speed routing fabric can receive the most significant part of an address address and possibly start making a routing decision before the other bytes of the frame have arrived. Hands down no brainer. Even the fact that headers come before payloads is a kind of "big endian", as is the fact that important information tends to occur earlier in headers. Look at a basic Ethernet frame. T…

Maybe that mattered on serial connections, but on a 10Gbit switch you're talking nonsense.

No, cut-through mode for packet forwarding (including on layer 3 prefix matches!) is a feature in current use on L2/3 10/40/100Gb switches, primarily in HFT/HPC environments.

Re: Ask HN: Is big-endian dead?

#64

Post by kazinator »

phs2501 wrote: post #18
jfries wrote: post #11

Little-endian is slightly more confusing for humans I've heard this before, but the reason is that you view hex data and list numbers left-to-right as if they were letters. They are not. 0x12345678 stored big-endian, numbering bytes left-to-right: 12 34 56 78 Looks good, but I think that this is actually more confusing, because when you number the bytes and bits you will see that the bytes are written left-to-right,…

At least in Freescale's PowerPC documentation, it's convention to number the bits left-to-right in big-endian. So the most-significant bit is bit 0, which matches up with the most-significant big-endian byte being 0. See, for a random example, page 1101 of https://www.nxp.com/docs/en/reference-manual/MPC8379ERM.pdf . Personally I prefer the little-endian representation.

The documentation might have that numbering, but that makes no difference in programming on the PowerPC. When we take the value 1 and shift left by 1 bit, we get 2.

That the documentation thinks this is bit 7 going to bit 6 is immaterial.

Calling the MSB "bit 1" is a tip of the hat to serial communications. In serial communication and networking, it is predominant to transmit the MSB first.

If the documentation is about a wire format, then using that numbering is correct down to the data link layer and (modulo framing considerations and such), physical.

Re: Ask HN: Is big-endian dead?

#66

Post by astrodust »

dogecoinbase wrote: post #63
astrodust wrote: post #58

Earlier quoted context omitted.

Maybe that mattered on serial connections, but on a 10Gbit switch you're talking nonsense.

No, cut-through mode for packet forwarding (including on layer 3 prefix matches!) is a feature in current use on L2/3 10/40/100Gb switches, primarily in HFT/HPC environments.

It's my understanding that this works with the complete address, not just the first byte, where it begins processing before the whole frame is received.

On 100Gbit you're talking 0.01ns per bit, so 0.32ns for a full IPv4 address. This compared to, potentially, 15.24ns for a complete IPv4 frame.

Re: Ask HN: Is big-endian dead?

#67

Post by api »

bitwize wrote: post #61
tlb wrote: post #4

Earlier quoted context omitted.

Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. You can even deliver those files to web browsers and use JS's TypedArray to get random access into them. (That requires a bit more than simply little-endian. It requires struct alignment and floating point formats to be the same. But with only a…

> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen. The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you defi…

Most compilers have directives for packing structs. What the OP suggests is something that can be done for high performance, but don't expect it to be portable outside of common architectures and compilers. I'd consider it something that shouldn't be done unless you really need the simplicity or performance.

Re: Ask HN: Is big-endian dead?

#68

Post by Gibbon1 »

rdtsc wrote: post #51

As long as we use the traditional network protocols and socket API it's not "dead" dead. The other name for big-endian is "network order", after all. As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing. In one instance I was reading a spec for an industry-specific protocol. At…

> As long as we use the traditional network protocols

I'm actually pissed the new LoRawan spec is big indian.

What saw that my only thought was 'dicks'

Re: Ask HN: Is big-endian dead?

#69

Post by kazinator »

astrodust wrote: post #58
kazinator wrote: post #48

Earlier quoted context omitted.

Big-endian means that am ultra-high-speed routing fabric can receive the most significant part of an address address and possibly start making a routing decision before the other bytes of the frame have arrived. Hands down no brainer. Even the fact that headers come before payloads is a kind of "big endian", as is the fact that important information tends to occur earlier in headers. Look at a basic Ethernet frame. T…

Maybe that mattered on serial connections, but on a 10Gbit switch you're talking nonsense.

Only if all you care about is that 10Gbit is a lot of throughput, and latencies don't bother your application.

If a 10Gbit network is being used precisely because 1Gbit didn't have sufficiently low latency, then it matters.

Re: Ask HN: Is big-endian dead?

#70

Post by scott00 »

rdtsc wrote: post #51

As long as we use the traditional network protocols and socket API it's not "dead" dead. The other name for big-endian is "network order", after all. As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing. In one instance I was reading a spec for an industry-specific protocol. At…

Using the x86-64 C struct layout as the serialization format is increasingly common in financial trading protocols. Skipping all the bit twiddling has a huge performance impact at high message rates. I once compared two feeds covering the same basic data, where the major difference was one had a complicated serialization and one was C struct layout. The former needed two 16 core servers just to process I/O and deserialize the messages; the latter used 1 core on a single server.
Post reply on HN