Live data from Hacker News

Ask HN: Is big-endian dead?

news.ycombinator.com

151–160 of 193 posts

Re: Ask HN: Is big-endian dead?

#151
post #114

Earlier quoted context omitted.

> The other name for big-endian is "network order", after all. I wonder where this started... I always suspected it was Sun with their "The Network is the Computer" motto, and of course they would define "network byte order" as what they used. Today, it drives me crazy that we're constantly swapping before sending across the network and then swapping again when it's received.

This may be the best ensemble of answers you'll get to that question: https://retrocomputing.stackexchange.com/questions/2652/when... The internet was big-endian from the start, and that was probably because DEC machines were big-endian.

DEC's PDP-11, VAX, and Alpha were all little-endian. Alpha was biendian but Cray Research was the only vendor that shipped a big endian configuration.

Re: Ask HN: Is big-endian dead?

#152
post #133

Earlier quoted context omitted.

Baaad idea. Quassel standardized years ago on Qt’s data serialization everywhere, and it’s become a major issue now.

Story time, please.

I only started contributing in '14, so most of what I heard is second hand information, this is all simpkified, and some parts will likely be wrong. It's also all my own opinion, I'm not representing any project here.

Basically, a decade ago a student started to work with some friends on an IRC client that integrates with a custom bouncer. Being a prototype, they just used Qt's serialization protocol between them. Over the years the project grew, at some point nokia funded development, it became Kubuntu's default IRC client (but only in the client+bouncer in one binary version, so without all the advantages), nokia was bought and closed and the department sold to BMW, and at some point.

Now, people tried writing third-party clients for this. And this became a minor issue, because the protocol was never documented. In favt, Qt's serialization was used for storing configs on disk, some blobs in the database, and over the network. It was later wrapped in TLS and deflate, and even at some point array-of-struct was turned into struct-of-array for the pattern during initialisation.

Either way, someone tried writing a mobile client for it, decided the protocol was insane, and instead built his own, almost identical client/bouncer system with an Erlang backend and json as protocol. This grew, and became IRCCloud.

Now, other people again tried developing third party clients for quassel. An android client was developed, but development was messy, and over the years, it stalled, because they reverse engineered the protocol, semi-successfully, and at some point didn't have enough time left, and gave up.

People reversed the protocol partially over the years again for pyquassel and quasselc/quasselbots/quassel-irssi.

Around then, another person tried reversing the protocol, and reimplementing it in JS for a webclient, which after a while became quassel-webserver.

Back then there was a lot of talk about replacing the protocol, but it was never done yet.

I, a user of irccloud around then, was annoyed with the costs (still being in high school myself, I couldn't afford the 4$/month, and the free tier wasn't enough), so I out for alternatives, and found quassel. But I hated the looks of Quasseldroid, so I forked it, and started working on the UI, and on features by reversing the protocol yet again. Not having actually programmed anything except for some delphi and VB.NET stuff, a tiny java project and one C# Windows Phone 7 app, my code was the worst, ever. Seriously, it was bad. After a while, discussion came about about turning this code into a PR, so I threw it all away, and rewrote it again, still bad, but it worked. This was merged, I became maintainer of Quasseldroid (because no one else was working on it anymore), and then, around 2015, I reversed the protocol, read the entire source of every implementation, wrote it all down on paper, studied every file format and every quirks and then I rewrote quasseldroid from scratch, in about 3 months, with every feature of the desktop version. I called this The Next Generation of Quasseldroid, jokingly quasseldroid TNG or later quasseldroid-ng.

A few weeks later, a new Android version came out, introducing Doze, and breaking everything about quasseldroid-ng. And so I rewrote it again, and before release, a new Android version broke it all again.

And that basically repeated, until I decided that it can't continue like this, and if we'll do major changes to the protocol, we might actually get a working version for Android that lasts longer than a few weeks in Beta before Google breaks it. So I learnt C++, and started contributing.

And that is basically my view of the story. Reverse engineering a protocol again and again, seeing variations of variations of the same protocol everywhere used, never properly documented. The Qt documentation is entirely different from what Qt actually puts on the wire. Blobs in the database.

But it also means backwards compatibility in the desktop client/core for every version from the past 10 years, and backwards compatibility on Android for every version of the past 6 years.

And now, maybe, I'll be able to help replace this, bit by bit. After the improvements to tge protocol that added major performance benefits, the next part is replacing the bouncer-side config format entirely, so I can properly containerise it.

TL;DR: no matter how good the support for a non-standard binary serialization format in your favourite language is, 10 years down the road people will reuse your protocol in a dozen more languages, and they'll have to reverse the protocol themselves, and will do a semi-good job at it, and because you never thought about backwards compatibility you now have a mess (we had luck because 99% of what we transmitted were key/value maps, and when reading we always used default values if the key didn't exist and ignored unused keys. Sometimes we did serialize structs, basically, though, and those places still cause me headaches today, and require workarounds to add features, e.g. the latest sendermode implementations)

Re: Ask HN: Is big-endian dead?

#153
post #32

> Little-endian is slightly more confusing for humans... I think you meant to write “Little-endian is slightly more confusing for humans who use left-to-right languages ” as all the R-L languages also put the least significant digit on the right.

write out what happens bitwise with a shift on a little endian machine.

from a bytes as representing a bitstring perspective, big endian make sense, and little endian is a terrible scramble.

Re: Ask HN: Is big-endian dead?

#154

Eh, haswell added the movbe instructions. And I wouldn't be surprised if someone in the RISCV world added big endian instructions (they're nice for network processing). And nearly every powerpc I've seen is big endian, FWIW.

Pretty much all new deployment of Power that IBM has sold since the release of POWER8 is little endian. Distro support is very quickly shifting to be little endian only as well.

(disclaimer: IBMer)

Re: Ask HN: Is big-endian dead?

#155
post #18
post #11

Little-endian is slightly more confusing for humans I've heard this before, but the reason is that you view hex data and list numbers left-to-right as if they were letters. They are not. 0x12345678 stored big-endian, numbering bytes left-to-right: 12 34 56 78 Looks good, but I think that this is actually more confusing, because when you number the bytes and bits you will see that the bytes are written left-to-right,…

At least in Freescale's PowerPC documentation, it's convention to number the bits left-to-right in big-endian. So the most-significant bit is bit 0, which matches up with the most-significant big-endian byte being 0. See, for a random example, page 1101 of https://www.nxp.com/docs/en/reference-manual/MPC8379ERM.pdf . Personally I prefer the little-endian representation.

MSB0 is convention in basically all IBM documentation (including PowerPC/Power Architecture stuff), hence all the Freescale/NXP PPC manuals follow.

The main practical problem with MSB0 is that you need to consider the width of whatever field or register you're looking at to work out the correct bit shift.

Re: Ask HN: Is big-endian dead?

#156
post #124

Earlier quoted context omitted.

Not exctly accurate, for the part of the number that is Also for the part that is L-R, that is not the rule, as some people still read all the number as R-L (actually in a lot of historical documents that was the case), so the would read 1925 as five and twenty and nine hundred and a thousand. Where is now most people would read it as a thousand, and nine hundred, and five and twenty.

Many R-L languages do the same such as German French, Hindi et al.

Wait, Hindi is neither a R-L language nor does it switch directions for text and numbers.

Source: native Hindi speaker.

Re: Ask HN: Is big-endian dead?

#157
post #124

Earlier quoted context omitted.

Not exctly accurate, for the part of the number that is Also for the part that is L-R, that is not the rule, as some people still read all the number as R-L (actually in a lot of historical documents that was the case), so the would read 1925 as five and twenty and nine hundred and a thousand. Where is now most people would read it as a thousand, and nine hundred, and five and twenty.

Many R-L languages do the same such as German French, Hindi et al.

I guess you mean L-R. In Dutch and German 1925 is most commonly read as 'nine teen five and twenty' But 2025 as 'two thousand five and twenty'

Re: Ask HN: Is big-endian dead?

#159

Earlier quoted context omitted.

> Assuming little-endian CPU arch. It's followed by a byte reorder on big-endian architectures. Basically all common CPUs are LE. (And basically all BE CPUs have dedicated instructions for reading LE data. It's true I haven't yet added the inline assembly to use those instructions in Cap'n Proto's reference implementation, but that's only because no one actually cares about these architectures.) So in basically all r…

> but that's only because no one actually cares about these architectures. This. Optimizing performance on BE is a waste of everyone's time and resources.

Except, of course, for those small number of us who primarily do run on a BE platform. I don't mind doing this work, just let us do it.

Re: Ask HN: Is big-endian dead?

#160

Big endian is dead on the client. WebGL exposes endianness and happened at a time when almost all systems running browsers are little-endian. This makes the big-endian provisions of the spec a dead letter. Web devs don't need to test on big-endian systems and would have a hard time finding big-endian systems to test with even if they wanted to. It's safe to assume, therefore, that there's a mass of Web content that o…

You can not only assume it, but I can also say with certainty it exists. Emscripten hastened this by letting site owners compile little-endian code into little-endian asm.js. For example, the WhatsApp Web QR code generator won't run on a BE platform; it's an Emscriptenized blob of code backing it that assumes the typed array it stores into has LE orientation.

TenFourFox and Leopard Webkit both emulate little-endian typed arrays on BE Power Macs and transparently byteswap integers. This mostly works for sites that care and has almost no overhead in JITted code for 16 and 32-bit int, but Facebook managed to break some of our assumptions with floats (example: https://github.com/classilla/tenfourfox/issues/453 -- we don't byteswap floats because it's expensive and probably would break things expecting native endian on the DOM side). Fortunately, TenFourFox doesn't support WebGL, so we don't need to worry about that.

IMHO, this is a broken promise about how the Web was supposed to be platform-independent. I get the feeling the general attitude on this thread is that such platforms are unimportant, but that doesn't mean it's not a broken promise.

Post reply on HN