Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is big-endian dead?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 193 posts

Re: Ask HN: Is big-endian dead?

#81

Post by apaprocki »

rdtsc wrote: post #51

As long as we use the traditional network protocols and socket API it's not "dead" dead. The other name for big-endian is "network order", after all. As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing. In one instance I was reading a spec for an industry-specific protocol. At…

The same was done "back in the day". On our systems, we had a reliable transport protocol (precursor to TCP/IP) that took advantage of the fact that the network and hardware was all BE. Everyone tried to squeeze every last bit of performance out of the machines and avoiding the byte shuffling was not an insignificant performance gain. It's not surprising that people are now doing the same thing with x86-64 given its dominance.

Re: Ask HN: Is big-endian dead?

#82

Post by dasmoth »

astrodust wrote: post #57
gumby wrote: post #31

Earlier quoted context omitted.

FWIW every one of the machines you cited (“Sparc, old MIPS, old PPC, DEC Alpha, etc”) post dated the formation of the Internet (the arpanet transitioning to TCP); the Internet protocols just followed existing arpanet practice. Which was due to big-endian processors being common, but the dominant networked machine of that era was the PDP-10.

The PDP successor, the VAX, was strangely little-endian.

Not that strange — it was really the successor of the PDP-11, which was a quite different architecture.

Re: Ask HN: Is big-endian dead?

#83

Post by p-squared »

astrodust wrote: post #58
kazinator wrote: post #48

Earlier quoted context omitted.

Big-endian means that am ultra-high-speed routing fabric can receive the most significant part of an address address and possibly start making a routing decision before the other bytes of the frame have arrived. Hands down no brainer. Even the fact that headers come before payloads is a kind of "big endian", as is the fact that important information tends to occur earlier in headers. Look at a basic Ethernet frame. T…

Maybe that mattered on serial connections, but on a 10Gbit switch you're talking nonsense.

Yeah, it's extremely unlikely that a high-speed serial phy would even expose a primitive word size any smaller than 32 bits. Byte-by-byte decoding is extraordinarily difficult at these data rates.

Re: Ask HN: Is big-endian dead?

#84

Post by lazyjones »

drewg123 wrote: post #12
api wrote: post #3

Earlier quoted context omitted.

Big-endian is the canonical form for storage and the wire because back when the Internet was designed most "pro" machines were big-endian: Sparc, old MIPS, old PPC, DEC Alpha, etc. All these are dead or dying now. It's easy enough to deal with BE files and protocols by just swapping bytes. I'm referring to hardware architectures. Are there still any big-endian chips out there? Does it still make sense to support big-…

Small pedantic correction from a former Alpha kernel hacker: DEC Alpha was Big Endian only for some rare Cray systems. Everything else was Little Endian.

DECstations with MIPS CPU were also little-endian. Reason in both cases was probably VAX compatibility.

Re: Ask HN: Is big-endian dead?

#85

Post by apaprocki »

api wrote: post #3
tlb wrote: post #2

I used to use IETF's network order (big-endian) for everything on disk or over the wire, but I switched to little-endian around 2008. If I ever have to port to a big-endian machine I'll deal with it then. I haven't regretted it yet.

Big-endian is the canonical form for storage and the wire because back when the Internet was designed most "pro" machines were big-endian: Sparc, old MIPS, old PPC, DEC Alpha, etc. All these are dead or dying now. It's easy enough to deal with BE files and protocols by just swapping bytes. I'm referring to hardware architectures. Are there still any big-endian chips out there? Does it still make sense to support big-…

Just a small nit: the CPUs are all still actively developed and produced. For how much longer is anyone's guess. You shouldn't think of them as "very old systems". I run modern code happily on 64 CPU 4.3ghz POWER8 machines :)

Re: Ask HN: Is big-endian dead?

#86

Post by planteen »

comandillos wrote: post #59

Im working for an Space Agency and we'r using SPARC for all our sats CPUs. Basically, Big-Endian.

I worked on a code base using the LEON (SPARC, big-endian) and also ARM in little-endian mode for space. The code was common for both endians. I followed the sage advice of Rob Pike on it: https://commandcenter.blogspot.com/2012/04/byte-order-fallac...

Re: Ask HN: Is big-endian dead?

#87

Post by planteen »

bitwize wrote: post #61
tlb wrote: post #4

Earlier quoted context omitted.

Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. You can even deliver those files to web browsers and use JS's TypedArray to get random access into them. (That requires a bit more than simply little-endian. It requires struct alignment and floating point formats to be the same. But with only a…

> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen. The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you defi…

I totally agree with you and so does Rob Pike: https://commandcenter.blogspot.com/2012/04/byte-order-fallac...

Re: Ask HN: Is big-endian dead?

#88

Post by lazyjones »

valarauca1 wrote: post #26

Big Endian is still supported natively on POWER8/9 Intel added movBE (mov big endian) support in BM2 extension. Networks are still Big Endian. SPARC chips (and now RISC-V) are BIG Endian by default. MIPS just became an independent company a few days ago and they have native Big Endian support. Edit 1: I was wrong about Power8/9 LE/BE

ARM also supports both.

Re: Ask HN: Is big-endian dead?

#89

Post by bitwize »

IshKebab wrote: post #76
bitwize wrote: post #61

Earlier quoted context omitted.

> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen. The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you defi…

No, dumping internal representations to disk is orthogonal to zero-static data formats. You can have a well defined format that doesn't require byte-by-byte parsing. Look up FlatBuffers and CapnProto.

Both of those appear to work by deferring the parsing step to access time. You're still treating the thing as a bag of bytes and unpacking stuff out of it bytewise.

Re: Ask HN: Is big-endian dead?

#90

Post by phs2501 »

kazinator wrote: post #64
phs2501 wrote: post #18

Earlier quoted context omitted.

At least in Freescale's PowerPC documentation, it's convention to number the bits left-to-right in big-endian. So the most-significant bit is bit 0, which matches up with the most-significant big-endian byte being 0. See, for a random example, page 1101 of https://www.nxp.com/docs/en/reference-manual/MPC8379ERM.pdf . Personally I prefer the little-endian representation.

The documentation might have that numbering, but that makes no difference in programming on the PowerPC. When we take the value 1 and shift left by 1 bit, we get 2. That the documentation thinks this is bit 7 going to bit 6 is immaterial. Calling the MSB "bit 1" is a tip of the hat to serial communications. In serial communication and networking, it is predominant to transmit the MSB first. If the documentation is ab…

Oh, I'm aware it makes no difference internally; I picked that documentation because I am somewhat intimately familiar with it as I worked on a product with that CPU and did a lot of driver work. My only point was to the poster I was replying to, who wrote "because when you number the bytes and bits you will see that the bytes are written left-to-right, while at the same time the bits are written right-to-left". That assumes that bits are universally numbered starting with 0 at the LSB, which isn't true.
Post reply on HN