Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is big-endian dead?

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 193 posts

Re: Ask HN: Is big-endian dead?

#181

Post by kentonv »

classichasclass wrote: post #159
zerokernel wrote: post #128

Earlier quoted context omitted.

> but that's only because no one actually cares about these architectures. This. Optimizing performance on BE is a waste of everyone's time and resources.

Except, of course, for those small number of us who primarily do run on a BE platform. I don't mind doing this work, just let us do it.

FWIW I'd be very happy to accept patches adding the appropriate platform-specific inline assembly around here: https://github.com/capnproto/capnproto/blob/master/c++/src/c...

(But the status quo on BE is that it does a load followed by a byte swap, which is probably pretty cheap anyway. The compiler might even already know how to optimize that into the appropriate LE-load instruction.)

Re: Ask HN: Is big-endian dead?

#182

Post by kazinator »

amelius wrote: post #73
kazinator wrote: post #47

Earlier quoted context omitted.

OTOH, big-endian is better for comparison; you can tell when one integer is larger than another by looking at as little as just one bit, from the most significant end.

In the worst case, comparison is still like subtraction.

Even if the whole datum is examined, comparison is more parallelizable; it doesn't require carry propagation.

Re: Ask HN: Is big-endian dead?

#183

Post by krylon »

userbinator wrote: post #174
krylon wrote: post #162

Earlier quoted context omitted.

> they've designed the spec to be whatever GCC on x86-64 Linux machine would do to layout C the structures in memory. In a previous job, the software I worked on had a "file format" that just consisted of dumping a raw array of structs to disk. It was obviously pretty fast, but it made cringe a lot. What's worse, the compiler had flags to control if and how much struct padding to use, so reading a data file with a bi…

The ideal is probably no padding --- on x86, unaligned accesses basically need no extra cycles[1] and if it means structures shrink and reduce cache misses, could actually be better. [1] Unless you happen to access a field spanning two cache lines and miss, in which case the padded version would require accessing that second cacheilne anyway.

Yes!

Unfortunately, the default for the compiler we used[1] to align struct members on DWORD (32-bit) boundaries, for some technical reason I am sure was totally reasonable. And these guys all built their code with a special make-variable that appended the "don't-pad-structs-I-really-know-what-I-am-doing" flag to the compiler's command line.

I did read somewhere, that x86 (at least Pentium III and later) like to make memory reads from addresses that are a multiple of four. But the profiling data I was able to gather showed that that part of the application had a negligible impact on overall performance. Since the vague job description I had gotten said my job was to "make things faster", I decided not to look into this any further.

[1] OpenWatcom (http://www.openwatcom.org/)

Re: Ask HN: Is big-endian dead?

#184

Post by [deleted] »

simias wrote: post #53
BrandoElFollito wrote: post #43

Earlier quoted context omitted.

Do you mean that you read a text from the right to the left, and when you meet a number you jump to is leftmost digit, then read it LR, then jump agzin to its left (where there is a space) and resume reading text?

If you think about it in left-to-right languages reading numbers requires backtracking because you have to see how many digits there are to know if you're reading units, thousands, millions, billions... Take 12,345,567 for instance, if you want to read it aloud you first have to go all the way to the end to figure out that there are three groups of numbers, so the first one is million, then thousands, then units. If…

[deleted]

Re: Ask HN: Is big-endian dead?

#185

Post by Leftium »

jcl wrote: post #178
ajross wrote: post #8

Little endian is also better for multi-word addition/subtraction. Your first operand will be the low word, which your pointer conveniently already indexes. This was a classic trick in the days of 8 and 16 bit systems, and assembly code for modern modular crypto systems still uses it. Really I think consensus is that the industry focus on big endian systems was actually a mistake. It prioritized programmer comfort in…

Now that I think about it, most manual arithmetic would also be easier if we described numbers with the least-significant digits first. If you're doing sums in your head, you could just list out the answer's digits as you calculate them, rather than needing to remember the list of digits and reverse them at the end.

Alternate methods of manual arithmetic that calculate the most significant digits first eliminate the need to remember and reverse digits.

With practice it can be faster than entering on a calculator!

Re: Ask HN: Is big-endian dead?

#186

Post by Const-me »

bitwize wrote: post #177
Const-me wrote: post #121

Earlier quoted context omitted.

I have to disagree with your “No no no”. It all depends on requirements. When you’re on a PC, and working with files that you can reasonably expect will be exchanged (like doc, xls or pdf) — then your “No no no” heuristic is absolutely correct. As you pointed out, differences between compilers and between e.g. x86/amd64 are very likely to render these formats incompatible. But when you’re working with files that only…

> But when you’re working with files that only your software will access, unpacking it byte by byte will slow down IO by a huge factor compared to both mmap, and read/write of large blocks (the latter will likely translate to DMA i.e. the CPU will be free to do something else). If: * you're working with files that only your software will access AND * you control, and understand, the CPU architecture of any machine th…

> In general it's a good idea to err on the side of caution by default, profile, and then optimize the hot paths as necessary bearing the constraints you're assuming in mind

In general, changing data formats to something incompatible is one of the most expensive changes you can possibly make to your software. If you’re not sure write a prototype and profile. Implementing knowingly inefficient data format for a performance-critical application isn’t a good idea.

> it will only cause marginal harm to do everything in byte offsets and shift and OR bytes to yield final values

Huge harm, in both runtime performance, and code size & complexity.

P.S. For applications where portability matters and you’re OK paying performance cost of that, the industry has moved towards XML based formats. Not only it fixes byte order issues, also text encodings, globalization, it’s human readable, and it’s fast enough for many practical applications. E.g. doc/xls that you’ve mentioned are deprecated by docx/xlsx, the latter are XML based.

Re: Ask HN: Is big-endian dead?

#187

Post by asterius »

tlb wrote: post #103
kazinator wrote: post #69

Earlier quoted context omitted.

Only if all you care about is that 10Gbit is a lot of throughput, and latencies don't bother your application. If a 10Gbit network is being used precisely because 1Gbit didn't have sufficiently low latency, then it matters.

But more than that, the encoding used in 10GBASE-T doesn't allow decoding individual bytes. For the usual copper standard, it's 4 pairs with 7 bits per symbol, and the skew between lanes means you can't count on corresponding symbols arriving at the same time. And then there's a reed-solomon block code to correct errors, which has to be decoded all at once. The optical standards use 64b/66b coding, which means you re…

No one who cares about latency even has 10GBASE-T hardware, because it introduces a vast amount of coding latency. It is also very power inefficient.

Re: Ask HN: Is big-endian dead?

#188

Post by angry_octet »

While it has mostly gone out of favour, it really isn't due to some cosmic plan. Intel just won in the microprocessor architecture race, and they happened to be LE due to their evolution.

I still have PowerPC and MIPS machines that are BE, and which are occasionally handy for seeing if pointer math is being done wrong.

Re: Ask HN: Is big-endian dead?

#189

Post by ksec »

bonzini wrote: post #29
valarauca1 wrote: post #26

Big Endian is still supported natively on POWER8/9 Intel added movBE (mov big endian) support in BM2 extension. Networks are still Big Endian. SPARC chips (and now RISC-V) are BIG Endian by default. MIPS just became an independent company a few days ago and they have native Big Endian support. Edit 1: I was wrong about Power8/9 LE/BE

Recent Linux supports both endiannesses for PPC. ppc64le was added recently. RISC-V is little endian.

I thought RISC-V supported both?

Re: Ask HN: Is big-endian dead?

#190

Post by astrodust »

ycombobreaker wrote: post #144
astrodust wrote: post #58

Earlier quoted context omitted.

Maybe that mattered on serial connections, but on a 10Gbit switch you're talking nonsense.

It absolutely matters for low-latency applications. Maybe not your use case, but it's not nonsense.

It's nonsense when talking about bytes. As others have pointed out, most switches/routers work with chunks of 16-64 bytes at a time, not singles.
Post reply on HN